HEASARC Staff Scientist Position - Applications are now being accepted for a Staff Scientist with significant experience and interest in the technical aspects of astrophysics research, to work in the High Energy Astrophysics Science Archive Research Center (HEASARC) at NASA Goddard Space Flight Center (GSFC) in Greenbelt, MD. Refer to the AAS Job register for full details.

Search:

[Advanced Search]



RXTE Helpdesk/FAQ RXTE What's New HEASARC Site Map

RXTE
GOF		 XTE FITS Database Management for Users
Recipes from the RXTE Cook Book 		RXTE
FAQ

Introduction

The design of the XTE FITS Database (XFDB) is described in other documents with varying degrees of detail. This document aims to be a purely practical guide. It will cover three topics:

  1. The function of the Master Index (FMI) files and how XDF uses them - to provide background information for XDF and the next topic.

  2. How to create and fix FMI files - e.g. if you want to combine observations or build a database with multiple proposals and AOs, but don't have a combined FMI file.

  3. The calibration database (CALDB) - a brief introduction.

The ingredients used are: 

   xdf: XTE Data Finder, shipped with Ftools.

   $XFDB: Environment variable, to be set to the top of the database
      (the location of the FMI file).

   recofmi:  Tool to repair or create a higher level FMI  file.
The necessary tools and files can be downloaded via anonymous FTP from heasarc.gsfc.nasa.gov, in the directory xte/calib_data.

1. Master Index (FMI) Files

FMI files simply contain a list of observations, providing information like: 


   ObsId
   Source
   User
   RA and Dec
   Start and stop time
   Pointers to index files for all subsystems
Subsystem index files, in turn, contain pointers to the actual data files. Thus, XDF can locate any data file, as long as it is given the path to the directory where the FMI file resides.

Typically, the mission database contains FMI files at three levels: the full mission; a single proposal (in the "P" directories); and a single ObsId. Consequently, a proposal level FMI contains all the combined information of the FMI files in the underlying ObsId directories, while the full mission FMI contains all information from all proposal level FMI files underneath it.

Therefore, it does not make any difference whether XDF is pointed to the mission, proposal, or ObsId directory: as long as there is an FMI file, it will be able to locate the data files for that ObsId.

One should realize, though, that the file pointers in all index files are relative. If the FMI in ObsId directory 


   00023-02-01-00
points to PCA index 

   FIPC_3eb6dcb-3eb922d,
the reference in the proposal directory P00023 will be: 

   00023-02-01-00/FIPC_3eb6dcb-3eb922d,
and in the mission FMI: 

AO0/P00023/00023-02-01-00/FIPC_3eb6dcb-3eb922d.
One should take care not to move FMI files from one directory to another. On the other hand, moving entire directory trees does not affect FMI file in the underlying structure.

The best thing to do is to set the environment variable XFDB to the directory that you want to use as the top of the database; make sure that it contains an FMI file. In C shell: 


   setenv XFDB /data/kaah3/xteguest/xtegst1/data/P00023
XDF will use $XFDB as a first guess for its database path.

2. Repairing or Creating an FMI File

FMI files are crucial for XDF's working correctly. In some cases you may have to fix one or create a new one; for instance, if you have a number of observations from your proposal, but no up-to-date proposal level FMI file; or if you want to combine observations from different proposals.

The ftool for creating or updating FMI files is recofmi. The general usage is simple; for instance, to update an existing FMI file in proposal level directory P10408 after you've added observations which aren't listed in the current FMI: 


   recofmi dirpath=P10408
This reconstructs the file P10408/FMI according to the contents of FMI files in underlying directories. If level=1 (default), it will go only to the first level subdirectories (i.e. ObsIds in this case); if level=2, it will go only to the subdirectories two levels deep. Typically, level=1 is used in proposal directories to include the information from their ObsId directories; level=2 is used to incorporate information from proposal level FMI files into the mission database's FMI file.

Normally, rows are only added. If rows with identical ObsIds are found, the original is left in place. The option "overwrite=yes" will force such rows to be overwritten. The "delete=yes" option first wipes the FMI file clean and then fills it only with whatever ObsIds it finds in lower level FMI files (for instance, if you have only a few of the listed observations.) The "check=yes" option provides only a report and will not make any changes.

If you want to create a new FMI in a directory that does not have one (say you have some loose ObsIds and want to put them under on common directory), first copy an existing FMI file (it does not matter which) to that directory and then run "recofmi delete=yes" on it.

For a more complicated example, suppose you need to create a top level FMI file under the FD directory, which you will probably want to do once you have datasets from multiple proposals and AOs; you then simply add the "level=2" option. This will tell the tool to look two directory levels down (through AOs to the proposals) and find the information in those FMIs: 


    cd $XFDB
    cp AO1/P10033/FMI .
    recofmi dirpath=$XFDB level=2 delete=yes
In this case, recofmi will look in all AOn directories and all Pxxxxxx directories under those to find the proposal level FMIs. The resulting top level FMI will list all the observations from all AOs and proposals which are in your database.

3. The calibration database (CALDB)

The caltools are a subset of the FTOOLS which require installation of CALDB in order to find the proper files for their execution. Other XTE-specific tools use the CALDB as well, such as pcabackest and pcarsp.

In all but early AO1 processed data, your tape comes with the full database structure starting with FD as the top level directory containing an AOn directory and a Cal directory, as well as Clock and OrbitEphem. Under AOn are the proposal level directories, while the Cal directory contains all of the XTE calibration files. For early processing, your tape begins with the proposal level directory, and under each ObsId is a cal directory. Newer processing puts all of the calibration files in one higher level directory to save space, replacing the files in each ObsId with symbolic links.

If you have a question about RXTE, please send email to one of our help desks.

This page is maintained by the RXTE GOF and was last modified on Wednesday, 24-Aug-2022 11:10:30 EDT.