|
NOTICE:
This Legacy journal article was published in Volume 2, November 1992, and has not been
updated since publication. Please use the search facility above to find regularly-updated information about
this topic elsewhere on the HEASARC site.
|
The ROSAT Public Data
Archive
M. F. Corcoran, M. Duesterhaus,
and K. L. Rhode
HEASARC
1 Introduction
Most ROSAT observers know that the proprietary data period ends 54 weeks after
the data were mailed from the ROSAT Data Center. After expiration of this
proprietary period, the data enter the public domain. The ROSAT public
archive has been set up by the ROSAT project to enable access to public domain
PSPC and HRI datasets.
The ROSAT public archive is located at the Goddard Space Flight Center in the
US, at MPE in Germany and in the UK at Leicester University. In the US, storage
of and access to the data is accomplished via the National Space Science Data
Center's Data Archive and Distribution Service (NDADS). The US archive on NDADS
and the German and UK archives will contain a complete set of all public
observations. By mutual agreement between the US ROSAT Science Data Center
(USRSDC) and MPE, data will be archived in the format in which it was received
by the PI. This means that observations made by US observers and processed by
the USRSDC will enter the archive in PROS FITS format, while those observations
made by observers in Germany and the UK will enter the archive in EXSAS FITS
formats. Though these two data formats are significantly different, analysis
tools are available from the Guest Observer facilities in the US and Germany to
assist users in the analysis of data in either format. It is planned that
German processed data in the US archive will eventually be reprocessed into the
US format.
By international agreement, the ROSAT public archive will officially open on
November 1, 1992. In fact, users will have access to public data in the US
archive prior to this date. The following article describes how to identify,
request and retrieve data from the US archive. This article assumes the reader
to be familiar with the form and content of ROSAT data sets for the PSPC and
the HRI. Users unfamiliar with ROSAT data products should consult the ROSAT
data products guide, available by request from the US ROSAT Guest Observer
Facility (contact Karen Smale, 301-286-7612, ksmale@heasrc.gsfc.nasa.gov).
2 Accessing ROSAT Public Data
Access to data in the ROSAT public archive is a three step process. An archive
user must first identify the data set of interest. The data set must then be
requested from the archive. Finally, the dataset must be retrieved from the
archive and delivered to the user. Although the ROSAT public archive has the
capability to deliver data via almost any medium desired by the archive user
(magnetic tapes, CD, etc.), the preferred means of data request and transfer is
electronic via Internet or NSI-DECnet (formerly SPAN). In this article, we
discuss how users can identify and request data and electronically transfer the
data to their home machine.
2.1 Data identification
The first step to data access is to identify those public data sets which may
be of interest. The list of publicly available datasets can be queried in a
number of ways.
2.1.1 Via anonymous ftp
An ASCII text list of all public data sets is available via anonymous ftp from
heasarc.gsfc.nasa.gov (128.183.8.233). Users should connect to
legacy via anonymous ftp, then change directory to
DATA/rosat/pspc/doc for the lists of public PSPC data, or
DATA/rosat/hri/doc for lists of public HRI data. For the convenience
of users, this list appears in 3 forms: sorted by RA and DEC
(ppublic_data.pos, hpublic_data.pos for the PSPC, HRI
respectively), by sequence number (ppublic_data.seq,
hpublic_data.seq) and by date of public release
(ppublic_data.date, hpublic_data.date). These lists are
updated monthly to include the data which will enter the public archive in the
following month. Users can download these files and search them using standard
text editors. Note that each list is about 132 columns wide, so set the screen
width accordingly. A sample session is shown below.
NDADSB> ftp heasarc.gsfc.nasa.gov
ndadsb.gsfc.nasa.gov MultiNet FTP user process 3.0(102)
Connection opened (Assuming 8-bit connections)
<legacy FTP server (ULTRIX Version 4.1 Tue Mar 19 00:38:17 EST 1991)
ready.
HEASARC.GSFC.NASA.GOV>user anonymous
<Guest login ok, send ident as password.
Password:
<Guest login ok, access restrictions apply.
HEASARC.GSFC.NASA.GOV>cd DATA/rosat/pspc/doc
<CWD command successful.
HEASARC.GSFC.NASA.GOV>ls
<Opening data connection for /bin/ls (128.183.36.18,1269) (0
bytes).
ppublic_data.date
ppublic_data.pos
ppublic_data.seq
<Transfer complete.
HEASARC.GSFC.NASA.GOV> type ascii
HEASARC.GSFC.NASA.GOV> mget ppublic*
After logging on ("user anonymous") the user should give their e-mail
address as password. Change directory to the directory containing the lists
of data in the public archive ("cd DATA/rosat/pspc/doc"). Make sure
file type is set to ASCII ("type ascii") for the text files. Then
download the files to your home machine ("mget ppublic*" will transfer
all files starting with "ppublic" to your home machine).
2.1.2 Via MIPS
Potential archive users can use the ROSAT Mission Information and Planning
System (MIPS) to determine which US, UK or German data sets are in the public
archive. MIPS users should choose MISSION INFO from the MAIN
MENU, then DistribLists from the ROSAT MISSION INFORMATION
MENU. DistribLists is a list of all data sets distributed by
the USRSDC and MPE, including the date when the data are scheduled to go
public. It is important to note that the appearance of a dataset in
DistribLists does not guarantee that the data are actually available.
The dates in DistribLists give the planned date of release of the data
to the public archive; in general this date will be the same as the actual date
of release, but there may be some circumstances when the actual date of release
will differ from that given in DistribLists. MIPS allows users to
search DistribLists by position (ConeSearch), by object name
(NameSearch), by public release date (PubDateSearch), by
start time of the observation (ObsDateSearch), by PI name
(PISearch) and by sequence number (Sequence#Search). At the
successful conclusion of a search MIPS will allow the user to display details
about any of the observations which were found by MIPS. For more information
on the use of MIPS, consult the MIPS USERS GUIDE (available from Margo
Duesterhaus, 301-286-3569, duesterhaus@rosat.gsfc.nasa.gov)
2.1.3 Via BROWSE
Lists of all ROSAT data distributed by the USRSDC and MPE, and the dates when
the data become public, can be queried using the HEASARC BROWSE interface.
BROWSE allows users to search the lists by position, object name, PI name,
sequence number, start date, public release date, and more. Once an
observation of interest has been identified, BROWSE users can display details
about the observation.
For details about how to log in to the HEASARC On-line Service and access the
BROWSE databases, see the article in this issue or in Legacy #1 called
"The HEASARC On-line Service". Questions about BROWSE and requests for related
documentation should be sent to request@ndadsa.gsfc.nasa.gov.
2.2 Data Request
The fastest way to access ROSAT archival data from NDADS is via the Automated
Retrieval Mail System (ARMS). Users generate an e-mail request for data to
archives@ndadsa.gsfc.nasa.gov (Internet). This e-mail should have "request rosat data_type" as the subject,
while the body of the letter should contain the entry_id of the
dataset(s) of interest. Data_types and sample entry_ids
recognized by NDADS for the ROSAT project are given in Table 1. By specifying
a data_type, an archive user is given the opportunity to select a
subset of the files associated with an observation. This is useful because, as
ROSAT observers know, each ROSAT observation has associated with it a large
number of files, and typically not all of these files are used during standard
analysis. The correspondence between NDADS data_type and ROSAT files
is given in Table 2a and 2b for PSPC and HRI data processed by the USRSDC.
Data_types for data processed by MPE will be defined shortly.
Requests with a data_type of FITS will provide the archive
user with sufficient information for standard scientific analyses of most
observations from either instrument.
Two sample requests to ARMS generated from a VMS machine using NSI-DECnet
(formerly SPAN) and from a unix workstation using Internet are shown below. In
the examples, PROS-style FITS data from a single PSPC observation is requested.
However, a single e-mail request can ask for data from a number of files - the
only catch is that the data must all have the same data_type. Also,
it is important not to request very large amounts of data to avoid filling
the NDADS staging disk. The NDADS-imposed limit is about 10 Mbytes per
request.
If you want to request 2 distinct data_types (even if they are from
the same observation), you must submit 2 separate e-mail requests (or use the
ALL data_type).
Table 1. NDADS DATA TYPE Table
PROJECT DATA TYPE Actual Components ENTRY ID FORM
ROSAT FITS PROS-style FITS files rp110590
rf110587
rh110267
CONTENTS file containing a list of files same as above
archived with a given observation
BASIC Events and good times in FITS same as above
format
DERIVED images and derived products in same as above
FITS format
CALIBRATION calibration-related products in same as above
FITS format
ANCILLARY Orbit and housekeeping data in same as above
FITS format
ALL All files pertaining to a same as above
particular sequence #
ASCII TEXT output same as above
POSTSCRIPT images and plots in POSTSCRIPT same as above
format
2.2.1 VMS user via NSI-DECnet
An e-mail request should be sent to archives@ndadsa.gsfc.nasa.gov. The subject line should
follow the format
REQUEST ROSAT data_type
where
data_type = FITS, BASIC, DERIVED, CALIBRATION, ANCILLARY, ASCII,
POSTSCRIPT, CONTENTS, or ALL for data processed by the
USRSDC
The body of the e-mail should contain the entry_id of the dataset of interest,
where
entry_id = <p><i><ror_number>
where
<p> = r for US processed datasets
w for data processed by MPE
<i> = p for unfiltered PSPC
f for PSPC observations made with the Boron filter
h for HRI
<ror_number> = the 6 digit sequence number of the observation.
Thus a typical mail request to extract the available PROS-style FITS files from
the unfiltered PSPC pointing 110590 would look like:
MAIL> send
To: archives@ndadsa.gsfc.nasa.gov
Subj: request rosat fits
rp110590
<end of message>
2.2.2 Unix user via Internet
An e-mail request should be sent to archives@ndadsa.gsfc.nasa.gov . The subject
line should follow the format
request rosat data_type
where
data_type = FITS, BASIC, DERIVED, CALIBRATION, ANCILLARY, ASCII,
POSTSCRIPT, CONTENTS, or ALL for data processed by the
USRSDC
The body of the e-mail should contain the entry_id of the dataset of interest,
where
entry_id = <p><i><ror_number>
where
<p> = r for US processed datasets
w for data processed by MPE
<i> = p for unfiltered PSPC
f for PSPC observations made with the Boron filter
h for HRI
<ror_number> = the 6 digit sequence number of the observation.
Thus a typical mail request to extract the available PROS-style FITS files from
the unfiltered PSPC pointing 110590 would look like:
rosat% mail archives@ndadsa.gsfc.nasa.gov
subj: request rosat fits
rp110590
<end of message>
TABLE 2a. Data Type Description for US-Format PSPC Data
________________________________________________________________________________
NDADS HEASARC
DATA TYPE File description Sample file TYPE FORMAT
================================================================================
FITS photon list rp110590.fits photlist FITS
broad-band image rp110590_im1.fits image FITS
hard-band image rp110590_im2.fits image FITS
exposure map rp110590_mex.fits image FITS
soft-band image rp110590_im3.fits image FITS
SIMBAD source list rp110590_sky.fits table FITS
CONTENTS public contents file rp110590.public_contents ascii FITS
BASIC photon list rp110590.fits photlist FITS
master data stor rp110590.mds photlist FITS
DERIVED processing parameters rp110590.par table FITS
broad-band image rp110590_im1.fits image FITS
hard-band image rp110590_im2.fits image FITS
soft-band image rp110590_im3.fits image FITS
SIMBAD source list rp110590_sky.fits table FITS
CALIBRATION response matrix rp110590.dmp response FITS
instrument map rp110590.moi table FITS
Off-axis area rp110590.oar table FITS
radiation calibration rp110590.rcr table FITS
results
exposure map rp110590_mex.fits image FITS
ANCILLARY Aspect quality rp110590.asp auxdata FITS
corrected aspect rp110590.cas auxdata FITS
event rates rp110590.evr auxdata FITS
binned housekeeping rp110590.hkb auxdata FITS
split aspect rp110590.sa auxdata FITS smoothed aspect rp110590.sas auxdata FITS
split orbit rp110590.so auxdata FITS
housekeeping status rp110590.sta auxdata FITS
ASCII Extracted SIMBAD list rp110590_smx.asc ascii ASCII
V&V report rp110590_vv2924.report ascii ASCII
POSTSCRIPT aspect strip plots rp110590_aaq001_16.plt image PS
(1/obi)
event rate strip plot rp110590_evr001_16.plt image PS
(1/obi)
cover page rp110590_fps.plt image PS
housekeeping plot rp110590_hks001_16.plt image PS
(1/obi)
broad-band image rp110590_im1.plt image PS
hard-band image rp110590_im2.plt image PS
soft-band image rp110590_im3.plt image PS
SIMBAD source plot rp110590_smx.plt image PS
ALL all the above all the above
________________________________________________________________________________
TABLE 2b. Data Type Description for US-Format HRI Data
___________________________________________________________________________________
NDADS HEASARC
DATA TYPE File description Sample file TYPE FORMAT
================================================================================
FITS photon event list rh110267.fits photlist FITS
HRI image rh110267_img.fits image FITS
SIMBAD sky list rh110267_sky.fits table FITS
CONTENTS public contents file rh110267.public_contents ascii FITS
BASIC complete photon store rh110267.cps photlist FITS
photon event list rh110267.fits photlist FITS
merged status history rh110267.mob table FITS
split status history rh110267.ots table FITS
standard good intervals rh110267.sgi table FITS
standard screen results rh110267.ssc table FITS
temporal status history rh110267.tsh table FITS
DERIVED Processing parameters rh110267.anp table FITS
bright earth intervals rh110267.obi table FITS
bright earth event list rh110267.obt table FITS
Processing parameters rh110267.par table FITS
HRI image rh110267_img.fits image FITS
SIMBAD sky list rh110267_sky.fits table FITS
CALIBRATION charged particle rh110267.cpb table FITS
background
detector background rh110267.dbm table FITS
quantum efficiency map rh110267.qeg table FITS
ANCILLARY aspect histogram rh110267.ah table FITS
aspect offsets rh110267.ao table FITS
count rate histogram rh110267.dms table FITS
Science frame quality rh110267.htl auxdata FITS
live-time factors rh110267.ltf auxdata FITS
Observation angles rh110267.oan auxdata FITS
HRI sampling histogram rh110267.s1d table FITS
HRI SAA rates rh110267.saa auxdata FITS
split orbit rh110267.so auxdata FITS
science rates rh110267.sps auxdata FITS
POSTSCRIPT SIMBAD sky plot rh110267_sky.plt image PS
rates strip plot (1/obi) rh110267_str.plt image PS
SIMBAD source list rh110267_sky.asc ascii ASCII
ASCII V&V report rh110267_vv2925.report ascii ASCII
ALL all the above all the above
________________________________________________________________________________
2.3 Data retrieval
After receipt of the e-mail message sent by the user, ARMS will extract the
data to the NDADS staging area and send an e-mail message notifying the user
that the data are available. (In the case where ARMS did not find the
appropriate data on the optical disk, a notification to this effect would be
sent.) The ARMS notification lists all the files retrieved. Though retrieval
times will vary with usage, it is expected that in most cases ARMS can stage
the data within minutes of receipt of the e-mail request, giving users
"near-line" access to the data. The archive user can then retrieve the files
via anonymous ftp from ndadsa.gsfc.nasa.gov.
The basic steps are:
a) connect to NDADSA via anonymous ftp. Give your e-mail address as the
password.
b) change directory to ROSAT.
c) check the directory to make sure the expected files are there
d) To ensure the proper transmission of the binary data, the filetype must be
sent to binary whenever a FITS formatted file is being downloaded.
e) get the files.
A sample session is shown below:
$ ftp ndadsa
HEASRC.GSFC.NASA.GOV MultiNet FTP user process 3.0(102)
Connection opened (Assuming 8-bit connections)
<ndadsa.gsfc.nasa.gov MultiNet FTP Server Process 3.0(12) at Mon
28-Sep-92 5:35P
M-EDT
NDADSA.GSFC.NASA.GOV>user anonymous
<anonymous user ok. Send real ident as password.
Password:
<Guest User ROSAT_GUEST logged into ANON_DIR:[000000] at Mon 28-Sep-92
17:35, job 2
2001f6d.
<Directory and access restrictions apply
NDADSA.GSFC.NASA.GOV>cd rosat
<Connected to ANON_DIR:[000000.ROSAT].
NDADSA.GSFC.NASA.GOV>ls *.fits
<List started.
rp110590.fits
rp110590_im1.fits
rp110590_im2.fits
rp110590_im3.fits
rp110590_mex.fits
rp110590_sky.fits
<Transfer completed.
NDADSA.GSFC.NASA.GOV>type binary
Type: Image, Structure: VMS, Mode: Stream
NDADSA.GSFC.NASA.GOV>hash
NDADSA.GSFC.NASA.GOV>mget rp*.fits
NSI-DECnet users can also use VMS copy to transfer the files from
NDADSA::ANON_DIR:[ROSAT] to their home machine:
$ copy ndadsa::anon_dir:[rosat]rp*.fits */confirm/log
3 Other Issues
3.1 A word about multiple releases of data
A number of data sets have multiple release dates associated with them and thus
may appear in the MIPS and BROWSE lists more than once. Such instances will
occur if, for example, a proposed observation was made in two or more ROSAT
pointings separated by weeks or months, since, in order to provide the observer
with data in a timely fashion, the policy of the USRSDC has been to ship the
data from each pointing to the observer as soon processing is completed. As
the rest of the observation is made, the USRSDC processes those data, adds them
to the earlier data and sends a new shipment to the observer. In cases like
this, the policy of the ROSAT public archive is to release, at the appropriate
time, each data set sent to the PI by the USRSDC. This policy
allows the public greater access to data which may be scientifically useful,
without compromising the PI's data rights (since by definition an incomplete
observation is not adequate for the science proposed by the PI). It was also
decided that, as an observation is completed, the newer data will replace the
older data. Thus archive users in general will not be able to readily access
the earlier data sets if they have been superseded by a later, more complete
release.
Starting in November 1992, all data sets previously processed by the USRSDC
will be reprocessed by a new version of the processing software designed to
correct some relatively minor problems with the pre-November 1992 version. In
addition, public datasets which were processed by the new version of the
processing software will be archived in a new FITS format (the so-called
"rationalized" format), which is designed to be more user-friendly than the old
format. Data in the public archive in the "unrationalized" format will
gradually be replaced by data in the "rationalized" format. It is expected
that US, UK and German data in the rationalized FITS format, with associated
POSTSCRIPT and ASCII files, will constitute the official ROSAT public archive
in the US. Thus archive users in general will not be able to readily access
the "unrationalized" data sets if they have been superseded by a "rationalized"
version. However, data in the "unrationalized" format will be made available
to users by special request to NDADS.
There have been rare cases when corrupted data have been sent to a PI. In
these cases, data have been reprocessed and redistributed to the PI. It is the
policy of the ROSAT Public archive that the corrupt data will not enter the
archive. In these cases the official public release date for the data is 54
weeks after the "good" data were distributed by the USRSDC.
3.2 Future Events
The ROSAT public archive is still undergoing modifications. A number of new
features will be shortly available to the user community. These include
1) Use of BROWSE to display products extracted from the ROSAT public archive.
Users will be able to use BROWSE to display ROSAT images, spectra and light
curves and to perform rudimentary analyses. In addition, users will eventually
be able to use BROWSE to extract data products to their user account for
further analysis
2) Generating ARMS e-mail requests automatically using BROWSE via a
command-driven procedure.
3) Generating ARMS e-mail requests automatically using MIPS.
Since MIPS will use an easy-to-fill-out screen form, MIPS will probably provide
the easiest means by which users can request data from NDADS.
4 Further Information
More detailed information about the ROSAT Public Archive can be obtained from
the ROSAT Archive User's Guide, which will be available shortly. Requests for
copies can be sent to Mike Corcoran at the address below. In addition,
periodic updates and announcements will be made in ROSAT e-mail bulletins and
the ROSAT Newsletter. Questions and/or comments related to the ROSAT
public archive should be addressed to:
Dr. Michael Corcoran
Code 668
Goddard Space Flight Center
Greenbelt MD 20771
(301) 286-5576
corcoran@barnegat.gsfc.nasa.gov
Proceed to the next article
Return to the previous article
Select another article
HEASARC Home |
Observatories |
Archive |
Calibration |
Software |
Tools |
Students/Teachers/Public
Last modified: Wednesday, 20-Oct-2021 10:51:50 EDT
HEASARC Staff Scientist Position - Applications are now being accepted for a Staff Scientist with significant experience and interest in the technical aspects of astrophysics research, to work in the High Energy Astrophysics Science Archive Research Center (HEASARC) at NASA Goddard Space Flight Center (GSFC) in Greenbelt, MD. Refer to the AAS Job register for full details.
| 
