NOTICE:

This Legacy journal article was published in Volume 2, November 1992, and has not been updated since publication. Please use the search facility above to find regularly-updated information about this topic elsewhere on the HEASARC site.

The ROSAT Public Data Archive

M. F. Corcoran, M. Duesterhaus,
and K. L. Rhode

HEASARC


1 Introduction

Most ROSAT observers know that the proprietary data period ends 54 weeks after the data were mailed from the ROSAT Data Center. After expiration of this proprietary period, the data enter the public domain. The ROSAT public archive has been set up by the ROSAT project to enable access to public domain PSPC and HRI datasets.

The ROSAT public archive is located at the Goddard Space Flight Center in the US, at MPE in Germany and in the UK at Leicester University. In the US, storage of and access to the data is accomplished via the National Space Science Data Center's Data Archive and Distribution Service (NDADS). The US archive on NDADS and the German and UK archives will contain a complete set of all public observations. By mutual agreement between the US ROSAT Science Data Center (USRSDC) and MPE, data will be archived in the format in which it was received by the PI. This means that observations made by US observers and processed by the USRSDC will enter the archive in PROS FITS format, while those observations made by observers in Germany and the UK will enter the archive in EXSAS FITS formats. Though these two data formats are significantly different, analysis tools are available from the Guest Observer facilities in the US and Germany to assist users in the analysis of data in either format. It is planned that German processed data in the US archive will eventually be reprocessed into the US format.

By international agreement, the ROSAT public archive will officially open on November 1, 1992. In fact, users will have access to public data in the US archive prior to this date. The following article describes how to identify, request and retrieve data from the US archive. This article assumes the reader to be familiar with the form and content of ROSAT data sets for the PSPC and the HRI. Users unfamiliar with ROSAT data products should consult the ROSAT data products guide, available by request from the US ROSAT Guest Observer Facility (contact Karen Smale, 301-286-7612, ksmale@heasrc.gsfc.nasa.gov).

2 Accessing ROSAT Public Data

Access to data in the ROSAT public archive is a three step process. An archive user must first identify the data set of interest. The data set must then be requested from the archive. Finally, the dataset must be retrieved from the archive and delivered to the user. Although the ROSAT public archive has the capability to deliver data via almost any medium desired by the archive user (magnetic tapes, CD, etc.), the preferred means of data request and transfer is electronic via Internet or NSI-DECnet (formerly SPAN). In this article, we discuss how users can identify and request data and electronically transfer the data to their home machine.

2.1 Data identification

The first step to data access is to identify those public data sets which may be of interest. The list of publicly available datasets can be queried in a number of ways.

2.1.1 Via anonymous ftp

An ASCII text list of all public data sets is available via anonymous ftp from legacy.gsfc.nasa.gov (128.183.8.233). Users should connect to legacy via anonymous ftp, then change directory to DATA/rosat/pspc/doc for the lists of public PSPC data, or DATA/rosat/hri/doc for lists of public HRI data. For the convenience of users, this list appears in 3 forms: sorted by RA and DEC (ppublic_data.pos, hpublic_data.pos for the PSPC, HRI respectively), by sequence number (ppublic_data.seq, hpublic_data.seq) and by date of public release (ppublic_data.date, hpublic_data.date). These lists are updated monthly to include the data which will enter the public archive in the following month. Users can download these files and search them using standard text editors. Note that each list is about 132 columns wide, so set the screen width accordingly. A sample session is shown below.

NDADSB> ftp legacy.gsfc.nasa.gov
ndadsb.gsfc.nasa.gov MultiNet FTP user process 3.0(102)
Connection opened (Assuming 8-bit connections)
<legacy FTP server (ULTRIX Version 4.1 Tue Mar 19 00:38:17 EST 1991) ready.
LEGACY.GSFC.NASA.GOV>user anonymous
<Guest login ok, send ident as password.
Password:
<Guest login ok, access restrictions apply.
LEGACY.GSFC.NASA.GOV>cd DATA/rosat/pspc/doc
<CWD command successful.
LEGACY.GSFC.NASA.GOV>ls
<Opening data connection for /bin/ls (128.183.36.18,1269) (0 bytes).
ppublic_data.date
ppublic_data.pos
ppublic_data.seq
<Transfer complete.
LEGACY.GSFC.NASA.GOV> type ascii
LEGACY.GSFC.NASA.GOV> mget ppublic*

After logging on ("user anonymous") the user should give their e-mail address as password. Change directory to the directory containing the lists of data in the public archive ("cd DATA/rosat/pspc/doc"). Make sure file type is set to ASCII ("type ascii") for the text files. Then download the files to your home machine ("mget ppublic*" will transfer all files starting with "ppublic" to your home machine).

2.1.2 Via MIPS

Potential archive users can use the ROSAT Mission Information and Planning System (MIPS) to determine which US, UK or German data sets are in the public archive. MIPS users should choose MISSION INFO from the MAIN MENU, then DistribLists from the ROSAT MISSION INFORMATION MENU. DistribLists is a list of all data sets distributed by the USRSDC and MPE, including the date when the data are scheduled to go public. It is important to note that the appearance of a dataset in DistribLists does not guarantee that the data are actually available. The dates in DistribLists give the planned date of release of the data to the public archive; in general this date will be the same as the actual date of release, but there may be some circumstances when the actual date of release will differ from that given in DistribLists. MIPS allows users to search DistribLists by position (ConeSearch), by object name (NameSearch), by public release date (PubDateSearch), by start time of the observation (ObsDateSearch), by PI name (PISearch) and by sequence number (Sequence#Search). At the successful conclusion of a search MIPS will allow the user to display details about any of the observations which were found by MIPS. For more information on the use of MIPS, consult the MIPS USERS GUIDE (available from Margo Duesterhaus, 301-286-3569, duesterhaus@rosat.gsfc.nasa.gov)

2.1.3 Via BROWSE

Lists of all ROSAT data distributed by the USRSDC and MPE, and the dates when the data become public, can be queried using the HEASARC BROWSE interface. BROWSE allows users to search the lists by position, object name, PI name, sequence number, start date, public release date, and more. Once an observation of interest has been identified, BROWSE users can display details about the observation.

For details about how to log in to the HEASARC On-line Service and access the BROWSE databases, see the article in this issue or in Legacy #1 called "The HEASARC On-line Service". Questions about BROWSE and requests for related documentation should be sent to request@ndadsa.gsfc.nasa.gov.

2.2 Data Request

The fastest way to access ROSAT archival data from NDADS is via the Automated Retrieval Mail System (ARMS). Users generate an e-mail request for data to archives@ndadsa.gsfc.nasa.gov (Internet). This e-mail should have "request rosat data_type" as the subject, while the body of the letter should contain the entry_id of the dataset(s) of interest. Data_types and sample entry_ids recognized by NDADS for the ROSAT project are given in Table 1. By specifying a data_type, an archive user is given the opportunity to select a subset of the files associated with an observation. This is useful because, as ROSAT observers know, each ROSAT observation has associated with it a large number of files, and typically not all of these files are used during standard analysis. The correspondence between NDADS data_type and ROSAT files is given in Table 2a and 2b for PSPC and HRI data processed by the USRSDC. Data_types for data processed by MPE will be defined shortly. Requests with a data_type of FITS will provide the archive user with sufficient information for standard scientific analyses of most observations from either instrument.

Two sample requests to ARMS generated from a VMS machine using NSI-DECnet (formerly SPAN) and from a unix workstation using Internet are shown below. In the examples, PROS-style FITS data from a single PSPC observation is requested. However, a single e-mail request can ask for data from a number of files - the only catch is that the data must all have the same data_type. Also, it is important not to request very large amounts of data to avoid filling the NDADS staging disk. The NDADS-imposed limit is about 10 Mbytes per request.

If you want to request 2 distinct data_types (even if they are from the same observation), you must submit 2 separate e-mail requests (or use the ALL data_type).

Table 1. NDADS DATA TYPE Table

PROJECT     DATA TYPE     Actual Components                  ENTRY ID FORM      

ROSAT       FITS          PROS-style FITS files              rp110590          
                                                             rf110587          
                                                             rh110267          
            CONTENTS      file containing a list of files    same as above     
       			  archived with a given observation
            BASIC         Events and good times in FITS      same as above
			  format
            DERIVED       images and derived products in     same as above     
			  FITS format
            CALIBRATION   calibration-related products in    same as above     
  			  FITS format                  
            ANCILLARY     Orbit and housekeeping data in     same as above
			  FITS format            
            ALL           All files pertaining to a          same as above     
                          particular sequence #  
            ASCII         TEXT output                        same as above     
            POSTSCRIPT    images and plots in POSTSCRIPT     same as above     
                          format 

2.2.1 VMS user via NSI-DECnet

An e-mail request should be sent to archives@ndadsa.gsfc.nasa.gov. The subject line should
follow the format 
	REQUEST ROSAT data_type
where 
	data_type =  FITS, BASIC, DERIVED, CALIBRATION, ANCILLARY, ASCII, 
		     POSTSCRIPT, CONTENTS, or ALL for data processed by the 
		     USRSDC  

The body of the e-mail should contain the entry_id of the dataset of interest,
where
	entry_id = <p><i><ror_number>
where
	<p> =	    	r 	for US processed datasets
		    	w	for data processed by MPE
	<i> =	    	p     	for unfiltered PSPC
	          	f    	for PSPC observations made with the Boron filter
	          	h     	for HRI
	<ror_number> = the 6 digit sequence number of the observation.

Thus a typical mail request to extract the available PROS-style FITS files from the unfiltered PSPC pointing 110590 would look like:

MAIL> send
To:     archives@ndadsa.gsfc.nasa.gov
Subj:   request rosat fits

rp110590
<end of message>

2.2.2 Unix user via Internet

An e-mail request should be sent to archives@ndadsa.gsfc.nasa.gov . The subject line should follow the format

	request rosat data_type
where 
	data_type = FITS, BASIC, DERIVED, CALIBRATION, ANCILLARY, ASCII, 
		    POSTSCRIPT, CONTENTS, or ALL for data processed by the 
		    USRSDC  

The body of the e-mail should contain the entry_id of the dataset of interest,
where
	entry_id = <p><i><ror_number>
where
	<p> =	r 	for US processed datasets
		w	for data processed by MPE
	<i> =	p     	for unfiltered PSPC
	        f    	for PSPC observations made with the Boron filter
	        h     	for HRI
	<ror_number> = the 6 digit sequence number of the observation.

Thus a typical mail request to extract the available PROS-style FITS files from the unfiltered PSPC pointing 110590 would look like:

rosat% mail archives@ndadsa.gsfc.nasa.gov
subj: request rosat fits
rp110590
<end of message>

TABLE 2a. Data Type Description for US-Format PSPC Data

________________________________________________________________________________
NDADS                                                                    HEASARC                     
DATA TYPE    File description        Sample file               TYPE      FORMAT        
================================================================================
FITS         photon list             rp110590.fits             photlist  FITS          
             broad-band image        rp110590_im1.fits         image     FITS          
             hard-band image         rp110590_im2.fits         image     FITS          
             exposure map            rp110590_mex.fits         image     FITS          
             soft-band image         rp110590_im3.fits         image     FITS          
             SIMBAD source list      rp110590_sky.fits         table     FITS  
CONTENTS     public contents file    rp110590.public_contents  ascii     FITS          
BASIC        photon list             rp110590.fits             photlist  FITS          
             master data stor        rp110590.mds              photlist  FITS          
DERIVED      processing parameters   rp110590.par              table     FITS         
             broad-band image        rp110590_im1.fits         image     FITS          
             hard-band image         rp110590_im2.fits         image     FITS          
             soft-band image         rp110590_im3.fits         image     FITS          
             SIMBAD source list      rp110590_sky.fits         table     FITS          
CALIBRATION  response matrix         rp110590.dmp              response  FITS          
             instrument map          rp110590.moi              table     FITS          
             Off-axis area           rp110590.oar              table     FITS          
             radiation calibration   rp110590.rcr              table     FITS          
             results                                                            
             exposure map            rp110590_mex.fits         image     FITS          
ANCILLARY    Aspect quality          rp110590.asp              auxdata   FITS          
             corrected aspect        rp110590.cas              auxdata   FITS          
             event rates             rp110590.evr              auxdata   FITS          
             binned housekeeping     rp110590.hkb              auxdata   FITS          
             split aspect            rp110590.sa               auxdata   FITS                         smoothed aspect            rp110590.sas                      auxdata       FITS          
             split orbit             rp110590.so               auxdata   FITS
             housekeeping status     rp110590.sta              auxdata   FITS
ASCII        Extracted SIMBAD list   rp110590_smx.asc          ascii     ASCII
             V&V report	       	     rp110590_vv2924.report    ascii     ASCII         
POSTSCRIPT   aspect strip plots      rp110590_aaq001_16.plt    image     PS
             (1/obi)                                                   
             event rate strip plot   rp110590_evr001_16.plt    image     PS
             (1/obi)                                                 
             cover page              rp110590_fps.plt          image     PS
             housekeeping plot       rp110590_hks001_16.plt    image     PS
	     (1/obi)
             broad-band image        rp110590_im1.plt          image     PS
             hard-band image         rp110590_im2.plt          image     PS
             soft-band image         rp110590_im3.plt          image     PS
             SIMBAD source plot      rp110590_smx.plt          image     PS
ALL          all the above           all the above                                                 
________________________________________________________________________________

TABLE 2b. Data Type Description for US-Format HRI Data

___________________________________________________________________________________
NDADS                                                            HEASARC  
DATA TYPE    File description          Sample file               TYPE      FORMAT        
================================================================================
FITS         photon event list         rh110267.fits             photlist  FITS          
             HRI image                 rh110267_img.fits         image     FITS          
             SIMBAD sky list           rh110267_sky.fits         table     FITS          
CONTENTS     public contents file      rh110267.public_contents  ascii     FITS          
BASIC        complete photon store     rh110267.cps              photlist  FITS          
             photon event list         rh110267.fits             photlist  FITS          
             merged status history     rh110267.mob              table     FITS          
             split status history      rh110267.ots              table     FITS          
             standard good intervals   rh110267.sgi              table     FITS          
             standard screen results   rh110267.ssc              table     FITS         
             temporal status history   rh110267.tsh              table     FITS          
DERIVED      Processing parameters     rh110267.anp              table     FITS          
             bright earth intervals    rh110267.obi              table     FITS          
             bright earth event list   rh110267.obt              table     FITS          
             Processing parameters     rh110267.par              table     FITS         
             HRI image                 rh110267_img.fits         image     FITS          
             SIMBAD sky list           rh110267_sky.fits         table     FITS          
CALIBRATION  charged particle          rh110267.cpb              table     FITS          
             background                                                      
             detector background       rh110267.dbm              table     FITS          
             quantum efficiency map    rh110267.qeg              table     FITS          
ANCILLARY    aspect histogram          rh110267.ah               table     FITS          
             aspect offsets            rh110267.ao               table     FITS          
             count rate histogram      rh110267.dms              table     FITS          
             Science frame quality     rh110267.htl              auxdata   FITS          
             live-time factors         rh110267.ltf              auxdata   FITS          
             Observation angles        rh110267.oan              auxdata   FITS          
             HRI sampling histogram    rh110267.s1d              table     FITS          
             HRI SAA rates             rh110267.saa              auxdata   FITS          
             split orbit               rh110267.so               auxdata   FITS          
             science rates             rh110267.sps              auxdata   FITS          
POSTSCRIPT   SIMBAD sky plot           rh110267_sky.plt          image     PS            
             rates strip plot (1/obi)  rh110267_str.plt          image     PS            
             SIMBAD source list        rh110267_sky.asc          ascii     ASCII         
ASCII        V&V report                rh110267_vv2925.report    ascii     ASCII         
ALL          all the above             all the above                           
________________________________________________________________________________

2.3 Data retrieval

After receipt of the e-mail message sent by the user, ARMS will extract the data to the NDADS staging area and send an e-mail message notifying the user that the data are available. (In the case where ARMS did not find the appropriate data on the optical disk, a notification to this effect would be sent.) The ARMS notification lists all the files retrieved. Though retrieval times will vary with usage, it is expected that in most cases ARMS can stage the data within minutes of receipt of the e-mail request, giving users "near-line" access to the data. The archive user can then retrieve the files via anonymous ftp from ndadsa.gsfc.nasa.gov.

The basic steps are:
a) connect to NDADSA via anonymous ftp. Give your e-mail address as the password.
b) change directory to ROSAT.
c) check the directory to make sure the expected files are there
d) To ensure the proper transmission of the binary data, the filetype must be sent to binary whenever a FITS formatted file is being downloaded.
e) get the files.

A sample session is shown below:

$ ftp ndadsa
HEASRC.GSFC.NASA.GOV MultiNet FTP user process 3.0(102)
Connection opened (Assuming 8-bit connections)
<ndadsa.gsfc.nasa.gov MultiNet FTP Server Process 3.0(12) at Mon 28-Sep-92 5:35P
M-EDT
NDADSA.GSFC.NASA.GOV>user anonymous
<anonymous user ok. Send real ident as password.
Password:
<Guest User ROSAT_GUEST logged into ANON_DIR:[000000] at Mon 28-Sep-92 17:35, job 2
2001f6d.
<Directory and access restrictions apply
NDADSA.GSFC.NASA.GOV>cd rosat
<Connected to ANON_DIR:[000000.ROSAT].
NDADSA.GSFC.NASA.GOV>ls *.fits
<List started.
rp110590.fits
rp110590_im1.fits
rp110590_im2.fits
rp110590_im3.fits
rp110590_mex.fits
rp110590_sky.fits
<Transfer completed.
NDADSA.GSFC.NASA.GOV>type binary
Type: Image, Structure: VMS, Mode: Stream
NDADSA.GSFC.NASA.GOV>hash
NDADSA.GSFC.NASA.GOV>mget rp*.fits

NSI-DECnet users can also use VMS copy to transfer the files from NDADSA::ANON_DIR:[ROSAT] to their home machine:

$ copy ndadsa::anon_dir:[rosat]rp*.fits */confirm/log

3 Other Issues

3.1 A word about multiple releases of data

A number of data sets have multiple release dates associated with them and thus may appear in the MIPS and BROWSE lists more than once. Such instances will occur if, for example, a proposed observation was made in two or more ROSAT pointings separated by weeks or months, since, in order to provide the observer with data in a timely fashion, the policy of the USRSDC has been to ship the data from each pointing to the observer as soon processing is completed. As the rest of the observation is made, the USRSDC processes those data, adds them to the earlier data and sends a new shipment to the observer. In cases like this, the policy of the ROSAT public archive is to release, at the appropriate time, each data set sent to the PI by the USRSDC. This policy allows the public greater access to data which may be scientifically useful, without compromising the PI's data rights (since by definition an incomplete observation is not adequate for the science proposed by the PI). It was also decided that, as an observation is completed, the newer data will replace the older data. Thus archive users in general will not be able to readily access the earlier data sets if they have been superseded by a later, more complete release.

Starting in November 1992, all data sets previously processed by the USRSDC will be reprocessed by a new version of the processing software designed to correct some relatively minor problems with the pre-November 1992 version. In addition, public datasets which were processed by the new version of the processing software will be archived in a new FITS format (the so-called "rationalized" format), which is designed to be more user-friendly than the old format. Data in the public archive in the "unrationalized" format will gradually be replaced by data in the "rationalized" format. It is expected that US, UK and German data in the rationalized FITS format, with associated POSTSCRIPT and ASCII files, will constitute the official ROSAT public archive in the US. Thus archive users in general will not be able to readily access the "unrationalized" data sets if they have been superseded by a "rationalized" version. However, data in the "unrationalized" format will be made available to users by special request to NDADS.

There have been rare cases when corrupted data have been sent to a PI. In these cases, data have been reprocessed and redistributed to the PI. It is the policy of the ROSAT Public archive that the corrupt data will not enter the archive. In these cases the official public release date for the data is 54 weeks after the "good" data were distributed by the USRSDC.

3.2 Future Events

The ROSAT public archive is still undergoing modifications. A number of new features will be shortly available to the user community. These include

1) Use of BROWSE to display products extracted from the ROSAT public archive.
Users will be able to use BROWSE to display ROSAT images, spectra and light curves and to perform rudimentary analyses. In addition, users will eventually be able to use BROWSE to extract data products to their user account for further analysis

2) Generating ARMS e-mail requests automatically using BROWSE via a command-driven procedure.

3) Generating ARMS e-mail requests automatically using MIPS.

Since MIPS will use an easy-to-fill-out screen form, MIPS will probably provide the easiest means by which users can request data from NDADS.

4 Further Information

More detailed information about the ROSAT Public Archive can be obtained from the ROSAT Archive User's Guide, which will be available shortly. Requests for copies can be sent to Mike Corcoran at the address below. In addition, periodic updates and announcements will be made in ROSAT e-mail bulletins and the ROSAT Newsletter. Questions and/or comments related to the ROSAT public archive should be addressed to:

Dr. Michael Corcoran
Code 668
Goddard Space Flight Center
Greenbelt MD 20771

(301) 286-5576

corcoran@barnegat.gsfc.nasa.gov


Next Proceed to the next article Previous Return to the previous article

Contents Select another article




HEASARC Home | Observatories | Archive | Calibration | Software | Tools | Students/Teachers/Public

Last modified: Monday, 19-Jun-2006 11:40:52 EDT