Due to the lapse in federal government funding, NASA is not updating this website. We sincerely regret this inconvenience.
NOTICE:This Legacy journal article was published in Volume 2, November 1992, and has not been updated since publication. Please use the search facility above to find regularly-updated information about this topic elsewhere on the HEASARC site. |
The ROSAT Public Data Archive
and K. L. Rhode
1 Introduction
Most ROSAT observers know that the proprietary data period ends 54 weeks after the data were mailed from the ROSAT Data Center. After expiration of this proprietary period, the data enter the public domain. The ROSAT public archive has been set up by the ROSAT project to enable access to public domain PSPC and HRI datasets.
The ROSAT public archive is located at the Goddard Space Flight Center in the US, at MPE in Germany and in the UK at Leicester University. In the US, storage of and access to the data is accomplished via the National Space Science Data Center's Data Archive and Distribution Service (NDADS). The US archive on NDADS and the German and UK archives will contain a complete set of all public observations. By mutual agreement between the US ROSAT Science Data Center (USRSDC) and MPE, data will be archived in the format in which it was received by the PI. This means that observations made by US observers and processed by the USRSDC will enter the archive in PROS FITS format, while those observations made by observers in Germany and the UK will enter the archive in EXSAS FITS formats. Though these two data formats are significantly different, analysis tools are available from the Guest Observer facilities in the US and Germany to assist users in the analysis of data in either format. It is planned that German processed data in the US archive will eventually be reprocessed into the US format.
By international agreement, the ROSAT public archive will officially open on November 1, 1992. In fact, users will have access to public data in the US archive prior to this date. The following article describes how to identify, request and retrieve data from the US archive. This article assumes the reader to be familiar with the form and content of ROSAT data sets for the PSPC and the HRI. Users unfamiliar with ROSAT data products should consult the ROSAT data products guide, available by request from the US ROSAT Guest Observer Facility (contact Karen Smale, 301-286-7612, ksmale@heasrc.gsfc.nasa.gov).
2 Accessing ROSAT Public Data
Access to data in the ROSAT public archive is a three step process. An archive user must first identify the data set of interest. The data set must then be requested from the archive. Finally, the dataset must be retrieved from the archive and delivered to the user. Although the ROSAT public archive has the capability to deliver data via almost any medium desired by the archive user (magnetic tapes, CD, etc.), the preferred means of data request and transfer is electronic via Internet or NSI-DECnet (formerly SPAN). In this article, we discuss how users can identify and request data and electronically transfer the data to their home machine.
2.1 Data identification
The first step to data access is to identify those public data sets which may be of interest. The list of publicly available datasets can be queried in a number of ways.
2.1.1 Via anonymous ftp
An ASCII text list of all public data sets is available via anonymous ftp from heasarc.gsfc.nasa.gov (128.183.8.233). Users should connect to legacy via anonymous ftp, then change directory to DATA/rosat/pspc/doc for the lists of public PSPC data, or DATA/rosat/hri/doc for lists of public HRI data. For the convenience of users, this list appears in 3 forms: sorted by RA and DEC (ppublic_data.pos, hpublic_data.pos for the PSPC, HRI respectively), by sequence number (ppublic_data.seq, hpublic_data.seq) and by date of public release (ppublic_data.date, hpublic_data.date). These lists are updated monthly to include the data which will enter the public archive in the following month. Users can download these files and search them using standard text editors. Note that each list is about 132 columns wide, so set the screen width accordingly. A sample session is shown below.
NDADSB> ftp heasarc.gsfc.nasa.gov
ndadsb.gsfc.nasa.gov MultiNet FTP user process 3.0(102)
Connection opened (Assuming 8-bit connections)
<legacy FTP server (ULTRIX Version 4.1 Tue Mar 19 00:38:17 EST 1991) ready.
HEASARC.GSFC.NASA.GOV>user anonymous
<Guest login ok, send ident as password.
Password:
<Guest login ok, access restrictions apply.
HEASARC.GSFC.NASA.GOV>cd DATA/rosat/pspc/doc
<CWD command successful.
HEASARC.GSFC.NASA.GOV>ls
<Opening data connection for /bin/ls (128.183.36.18,1269) (0 bytes).
ppublic_data.date
ppublic_data.pos
ppublic_data.seq
<Transfer complete.
HEASARC.GSFC.NASA.GOV> type ascii
HEASARC.GSFC.NASA.GOV> mget ppublic*
After logging on ("user anonymous") the user should give their e-mail address as password. Change directory to the directory containing the lists of data in the public archive ("cd DATA/rosat/pspc/doc"). Make sure file type is set to ASCII ("type ascii") for the text files. Then download the files to your home machine ("mget ppublic*" will transfer all files starting with "ppublic" to your home machine).
2.1.2 Via MIPS
Potential archive users can use the ROSAT Mission Information and Planning System (MIPS) to determine which US, UK or German data sets are in the public archive. MIPS users should choose MISSION INFO from the MAIN MENU, then DistribLists from the ROSAT MISSION INFORMATION MENU. DistribLists is a list of all data sets distributed by the USRSDC and MPE, including the date when the data are scheduled to go public. It is important to note that the appearance of a dataset in DistribLists does not guarantee that the data are actually available. The dates in DistribLists give the planned date of release of the data to the public archive; in general this date will be the same as the actual date of release, but there may be some circumstances when the actual date of release will differ from that given in DistribLists. MIPS allows users to search DistribLists by position (ConeSearch), by object name (NameSearch), by public release date (PubDateSearch), by start time of the observation (ObsDateSearch), by PI name (PISearch) and by sequence number (Sequence#Search). At the successful conclusion of a search MIPS will allow the user to display details about any of the observations which were found by MIPS. For more information on the use of MIPS, consult the MIPS USERS GUIDE (available from Margo Duesterhaus, 301-286-3569, duesterhaus@rosat.gsfc.nasa.gov)
2.1.3 Via BROWSE
Lists of all ROSAT data distributed by the USRSDC and MPE, and the dates when the data become public, can be queried using the HEASARC BROWSE interface. BROWSE allows users to search the lists by position, object name, PI name, sequence number, start date, public release date, and more. Once an observation of interest has been identified, BROWSE users can display details about the observation.
For details about how to log in to the HEASARC On-line Service and access the BROWSE databases, see the article in this issue or in Legacy #1 called "The HEASARC On-line Service". Questions about BROWSE and requests for related documentation should be sent to request@ndadsa.gsfc.nasa.gov.
2.2 Data Request
The fastest way to access ROSAT archival data from NDADS is via the Automated Retrieval Mail System (ARMS). Users generate an e-mail request for data to archives@ndadsa.gsfc.nasa.gov (Internet). This e-mail should have "request rosat data_type" as the subject, while the body of the letter should contain the entry_id of the dataset(s) of interest. Data_types and sample entry_ids recognized by NDADS for the ROSAT project are given in Table 1. By specifying a data_type, an archive user is given the opportunity to select a subset of the files associated with an observation. This is useful because, as ROSAT observers know, each ROSAT observation has associated with it a large number of files, and typically not all of these files are used during standard analysis. The correspondence between NDADS data_type and ROSAT files is given in Table 2a and 2b for PSPC and HRI data processed by the USRSDC. Data_types for data processed by MPE will be defined shortly. Requests with a data_type of FITS will provide the archive user with sufficient information for standard scientific analyses of most observations from either instrument.
Two sample requests to ARMS generated from a VMS machine using NSI-DECnet (formerly SPAN) and from a unix workstation using Internet are shown below. In the examples, PROS-style FITS data from a single PSPC observation is requested. However, a single e-mail request can ask for data from a number of files - the only catch is that the data must all have the same data_type. Also, it is important not to request very large amounts of data to avoid filling the NDADS staging disk. The NDADS-imposed limit is about 10 Mbytes per request.
If you want to request 2 distinct data_types (even if they are from the same observation), you must submit 2 separate e-mail requests (or use the ALL data_type).
PROJECT DATA TYPE Actual Components ENTRY ID FORM
ROSAT FITS PROS-style FITS files rp110590
rf110587
rh110267
CONTENTS file containing a list of files same as above
archived with a given observation
BASIC Events and good times in FITS same as above
format
DERIVED images and derived products in same as above
FITS format
CALIBRATION calibration-related products in same as above
FITS format
ANCILLARY Orbit and housekeeping data in same as above
FITS format
ALL All files pertaining to a same as above
particular sequence #
ASCII TEXT output same as above
POSTSCRIPT images and plots in POSTSCRIPT same as above
format
2.2.1 VMS user via NSI-DECnet
An e-mail request should be sent to archives@ndadsa.gsfc.nasa.gov. The subject line should
follow the format
REQUEST ROSAT data_type
where
data_type = FITS, BASIC, DERIVED, CALIBRATION, ANCILLARY, ASCII,
POSTSCRIPT, CONTENTS, or ALL for data processed by the
USRSDC
The body of the e-mail should contain the entry_id of the dataset of interest,
where
entry_id = <p><i><ror_number>
where
<p> = r for US processed datasets
w for data processed by MPE
<i> = p for unfiltered PSPC
f for PSPC observations made with the Boron filter
h for HRI
<ror_number> = the 6 digit sequence number of the observation.
Thus a typical mail request to extract the available PROS-style FITS files from the unfiltered PSPC pointing 110590 would look like:
MAIL> send To: archives@ndadsa.gsfc.nasa.gov Subj: request rosat fits rp110590 <end of message>
2.2.2 Unix user via Internet
An e-mail request should be sent to archives@ndadsa.gsfc.nasa.gov . The subject line should follow the format
request rosat data_type
where
data_type = FITS, BASIC, DERIVED, CALIBRATION, ANCILLARY, ASCII,
POSTSCRIPT, CONTENTS, or ALL for data processed by the
USRSDC
The body of the e-mail should contain the entry_id of the dataset of interest,
where
entry_id = <p><i><ror_number>
where
<p> = r for US processed datasets
w for data processed by MPE
<i> = p for unfiltered PSPC
f for PSPC observations made with the Boron filter
h for HRI
<ror_number> = the 6 digit sequence number of the observation.
Thus a typical mail request to extract the available PROS-style FITS files from the unfiltered PSPC pointing 110590 would look like:
rosat% mail archives@ndadsa.gsfc.nasa.gov
subj: request rosat fits
rp110590
<end of message>
________________________________________________________________________________
NDADS HEASARC
DATA TYPE File description Sample file TYPE FORMAT
================================================================================
FITS photon list rp110590.fits photlist FITS
broad-band image rp110590_im1.fits image FITS
hard-band image rp110590_im2.fits image FITS
exposure map rp110590_mex.fits image FITS
soft-band image rp110590_im3.fits image FITS
SIMBAD source list rp110590_sky.fits table FITS
CONTENTS public contents file rp110590.public_contents ascii FITS
BASIC photon list rp110590.fits photlist FITS
master data stor rp110590.mds photlist FITS
DERIVED processing parameters rp110590.par table FITS
broad-band image rp110590_im1.fits image FITS
hard-band image rp110590_im2.fits image FITS
soft-band image rp110590_im3.fits image FITS
SIMBAD source list rp110590_sky.fits table FITS
CALIBRATION response matrix rp110590.dmp response FITS
instrument map rp110590.moi table FITS
Off-axis area rp110590.oar table FITS
radiation calibration rp110590.rcr table FITS
results
exposure map rp110590_mex.fits image FITS
ANCILLARY Aspect quality rp110590.asp auxdata FITS
corrected aspect rp110590.cas auxdata FITS
event rates rp110590.evr auxdata FITS
binned housekeeping rp110590.hkb auxdata FITS
split aspect rp110590.sa auxdata FITS smoothed aspect rp110590.sas auxdata FITS
split orbit rp110590.so auxdata FITS
housekeeping status rp110590.sta auxdata FITS
ASCII Extracted SIMBAD list rp110590_smx.asc ascii ASCII
V&V report rp110590_vv2924.report ascii ASCII
POSTSCRIPT aspect strip plots rp110590_aaq001_16.plt image PS
(1/obi)
event rate strip plot rp110590_evr001_16.plt image PS
(1/obi)
cover page rp110590_fps.plt image PS
housekeeping plot rp110590_hks001_16.plt image PS
(1/obi)
broad-band image rp110590_im1.plt image PS
hard-band image rp110590_im2.plt image PS
soft-band image rp110590_im3.plt image PS
SIMBAD source plot rp110590_smx.plt image PS
ALL all the above all the above
________________________________________________________________________________
___________________________________________________________________________________
NDADS HEASARC
DATA TYPE File description Sample file TYPE FORMAT
================================================================================
FITS photon event list rh110267.fits photlist FITS
HRI image rh110267_img.fits image FITS
SIMBAD sky list rh110267_sky.fits table FITS
CONTENTS public contents file rh110267.public_contents ascii FITS
BASIC complete photon store rh110267.cps photlist FITS
photon event list rh110267.fits photlist FITS
merged status history rh110267.mob table FITS
split status history rh110267.ots table FITS
standard good intervals rh110267.sgi table FITS
standard screen results rh110267.ssc table FITS
temporal status history rh110267.tsh table FITS
DERIVED Processing parameters rh110267.anp table FITS
bright earth intervals rh110267.obi table FITS
bright earth event list rh110267.obt table FITS
Processing parameters rh110267.par table FITS
HRI image rh110267_img.fits image FITS
SIMBAD sky list rh110267_sky.fits table FITS
CALIBRATION charged particle rh110267.cpb table FITS
background
detector background rh110267.dbm table FITS
quantum efficiency map rh110267.qeg table FITS
ANCILLARY aspect histogram rh110267.ah table FITS
aspect offsets rh110267.ao table FITS
count rate histogram rh110267.dms table FITS
Science frame quality rh110267.htl auxdata FITS
live-time factors rh110267.ltf auxdata FITS
Observation angles rh110267.oan auxdata FITS
HRI sampling histogram rh110267.s1d table FITS
HRI SAA rates rh110267.saa auxdata FITS
split orbit rh110267.so auxdata FITS
science rates rh110267.sps auxdata FITS
POSTSCRIPT SIMBAD sky plot rh110267_sky.plt image PS
rates strip plot (1/obi) rh110267_str.plt image PS
SIMBAD source list rh110267_sky.asc ascii ASCII
ASCII V&V report rh110267_vv2925.report ascii ASCII
ALL all the above all the above
________________________________________________________________________________
2.3 Data retrieval
After receipt of the e-mail message sent by the user, ARMS will extract the data to the NDADS staging area and send an e-mail message notifying the user that the data are available. (In the case where ARMS did not find the appropriate data on the optical disk, a notification to this effect would be sent.) The ARMS notification lists all the files retrieved. Though retrieval times will vary with usage, it is expected that in most cases ARMS can stage the data within minutes of receipt of the e-mail request, giving users "near-line" access to the data. The archive user can then retrieve the files via anonymous ftp from ndadsa.gsfc.nasa.gov.
The basic steps are:
a) connect to NDADSA via anonymous ftp. Give your e-mail address as the password.
b) change directory to ROSAT.
c) check the directory to make sure the expected files are there
d) To ensure the proper transmission of the binary data, the filetype must be sent to binary whenever a FITS formatted file is being downloaded.
e) get the files.
A sample session is shown below:
$ ftp ndadsa
HEASRC.GSFC.NASA.GOV MultiNet FTP user process 3.0(102)
Connection opened (Assuming 8-bit connections)
<ndadsa.gsfc.nasa.gov MultiNet FTP Server Process 3.0(12) at Mon 28-Sep-92 5:35P
M-EDT
NDADSA.GSFC.NASA.GOV>user anonymous
<anonymous user ok. Send real ident as password.
Password:
<Guest User ROSAT_GUEST logged into ANON_DIR:[000000] at Mon 28-Sep-92 17:35, job 2
2001f6d.
<Directory and access restrictions apply
NDADSA.GSFC.NASA.GOV>cd rosat
<Connected to ANON_DIR:[000000.ROSAT].
NDADSA.GSFC.NASA.GOV>ls *.fits
<List started.
rp110590.fits
rp110590_im1.fits
rp110590_im2.fits
rp110590_im3.fits
rp110590_mex.fits
rp110590_sky.fits
<Transfer completed.
NDADSA.GSFC.NASA.GOV>type binary
Type: Image, Structure: VMS, Mode: Stream
NDADSA.GSFC.NASA.GOV>hash
NDADSA.GSFC.NASA.GOV>mget rp*.fits
NSI-DECnet users can also use VMS copy to transfer the files from NDADSA::ANON_DIR:[ROSAT] to their home machine:
$ copy ndadsa::anon_dir:[rosat]rp*.fits */confirm/log
3 Other Issues
3.1 A word about multiple releases of data
A number of data sets have multiple release dates associated with them and thus may appear in the MIPS and BROWSE lists more than once. Such instances will occur if, for example, a proposed observation was made in two or more ROSAT pointings separated by weeks or months, since, in order to provide the observer with data in a timely fashion, the policy of the USRSDC has been to ship the data from each pointing to the observer as soon processing is completed. As the rest of the observation is made, the USRSDC processes those data, adds them to the earlier data and sends a new shipment to the observer. In cases like this, the policy of the ROSAT public archive is to release, at the appropriate time, each data set sent to the PI by the USRSDC. This policy allows the public greater access to data which may be scientifically useful, without compromising the PI's data rights (since by definition an incomplete observation is not adequate for the science proposed by the PI). It was also decided that, as an observation is completed, the newer data will replace the older data. Thus archive users in general will not be able to readily access the earlier data sets if they have been superseded by a later, more complete release.
Starting in November 1992, all data sets previously processed by the USRSDC will be reprocessed by a new version of the processing software designed to correct some relatively minor problems with the pre-November 1992 version. In addition, public datasets which were processed by the new version of the processing software will be archived in a new FITS format (the so-called "rationalized" format), which is designed to be more user-friendly than the old format. Data in the public archive in the "unrationalized" format will gradually be replaced by data in the "rationalized" format. It is expected that US, UK and German data in the rationalized FITS format, with associated POSTSCRIPT and ASCII files, will constitute the official ROSAT public archive in the US. Thus archive users in general will not be able to readily access the "unrationalized" data sets if they have been superseded by a "rationalized" version. However, data in the "unrationalized" format will be made available to users by special request to NDADS.
There have been rare cases when corrupted data have been sent to a PI. In these cases, data have been reprocessed and redistributed to the PI. It is the policy of the ROSAT Public archive that the corrupt data will not enter the archive. In these cases the official public release date for the data is 54 weeks after the "good" data were distributed by the USRSDC.
3.2 Future Events
The ROSAT public archive is still undergoing modifications. A number of new features will be shortly available to the user community. These include
1) Use of BROWSE to display products extracted from the ROSAT public archive.
Users will be able to use BROWSE to display ROSAT images, spectra and light curves and to perform rudimentary analyses. In addition, users will eventually be able to use BROWSE to extract data products to their user account for further analysis
2) Generating ARMS e-mail requests automatically using BROWSE via a command-driven procedure.
3) Generating ARMS e-mail requests automatically using MIPS.
Since MIPS will use an easy-to-fill-out screen form, MIPS will probably provide the easiest means by which users can request data from NDADS.
4 Further Information
More detailed information about the ROSAT Public Archive can be obtained from the ROSAT Archive User's Guide, which will be available shortly. Requests for copies can be sent to Mike Corcoran at the address below. In addition, periodic updates and announcements will be made in ROSAT e-mail bulletins and the ROSAT Newsletter. Questions and/or comments related to the ROSAT public archive should be addressed to:
Dr. Michael Corcoran
Code 668
Goddard Space Flight Center
Greenbelt MD 20771
(301) 286-5576
corcoran@barnegat.gsfc.nasa.gov
Proceed to the next article 
Return to the previous article
Select another article