The Organization of the HEASARC Calibration Database

2005 Nov 28

Ian M George & Ron S Zellar
Code 668,
NASA/GSFC,
Greenbelt,
MD 20771


Version: 2005 Nov 28





SUMMARY

The location, organization and contents of the HEASARC calibration database on-line at NASA/GSFC is summarized.

LOG OF SIGNIFICANT CHANGES


Release Sections Changed Brief Notes
Date
1993 Apr 30 All First Public Version
1995 Feb 15 All Made compatible with LaTeX2HTML software
2004 Apr 01 All Made compatible with tth
2005 Nov 28 All Updated to reflect current state of FTP area and caldb structure

1  OVERVIEW

1.1  Location

The HEASARC calibration database (CALDB) is accessed via the legacy.gsfc.nasa.gov machine within the LHEA at NASA/GSFC: The entire caldb directory tree is available to remote users via anonymous ftp (see Section ). Within the HEASARC, the caldb has been remote mounted on all other machines and the symbolic links /caldb & /CALDB created to point to the above top level directory.

1.2  Basic Structure

The contents of the /caldb area are divided into the following sub-directories1:
All directories and files within the /caldb tree are freely accessible to remote users, and each is discussed in more detail in the following sections.

1.3  Relation to Other Areas of the HEASARC anonymous FTP account

During a reorganization of the HEASARC anonymous FTP account on legacy carried out in 1993 Feb, it was decided to re-organize the directory structure to facilitate the search for desired files. The ftp area contains the following sub-directories: with the structure described in some detail in Drake and O'Neel (1993).
There are two routes by which a user can access the calibration data: via /mission/calib_data, or via /caldb/mission. Many users may prefer to use the former route, collecting calibration data along with other information specific to a given mission/instrument. However users concerned only with calibration data (eg those maintaining their own, local copy of the CALDB) may prefer to use the latter route. It should be stressed however that both routes access the same physical files, and the two access routes are connected via a symbolic link (see Section ). A similar choice of routes also exists for some parts of the calibration software and documentation trees. These are also discussed in Sections  & respectively.

1.4  Access

As described in Section 1.1, at the present time the HEASARC CALDB exists on
ftp://legacy.gsfc.nasa.gov/caldb.

1.4.1  Anonymous ftp to the Legacy machine

Users of machines on the Internet network may use File Transfer Protocol (ftp), to list the contents and/or copy file from the HEASARC CALDB.
To initiate ftp from the command line, type the following (boldface) commands in a terminal window at the prompt (comments appear in italics):
ftp legacy.gsfc.nasa.gov then
ftp> user anonymous
Password: type your e-mail address as the password
You will now be in the top level directory2 of the HEASARC anonymous ftp account on ftp://legacy.gsfc.nasa.gov and a message to this effect, along with any important bulletins, should be displayed. A general overview of the account can be found in Drake and O'Neel (1993). More detailed on-line information, instructions etc concerning the ftp account can be found within the ASCII README file within this (and many other) directories.
The following list of basic ftp commands may be of use to the inexperienced ftp user:
ftp> help                 (or ftp> ?) displays list of ftp commands
ftp> help lcd displays help for ftp command lcd
ftp> cd caldb/docs move to the caldb/docs directory
ftp> cd ../ move back up one directory
ftp> ls brief list of the contents of the current legacy directory
ftp> dir detailed list of the contents of the current legacy directory
ftp> pwd display the present working directory on legacy
ftp> binary set binary transfer type (required for FITS files etc)
ftp> mget * copies all files in the current legacy directory to your area
ftp> ascii set ASCII transfer type (required for LATEXdocuments, postscript files etc)
ftp> get README copies the README from current Legacy dir to local dir
ftp> lcd /home change local directory to /home on user's home machine
ftp> quit quits from ftp & returns control to local operating system A variety of other useful commands are available within ftp, a number of which depend on which version of ftp is installed on the local machine.

2  CALIBRATION DATA FILES

The HEASARC CALDB classifies files as `Primary Calibration Files' `Basic Calibration Files' and `Calibration Product Files' (PCFs, BCFs & CPFs respectively; see also CAL/GEN/91-001). PCFs are 'raw' ground and in-orbit calibration datasets not of immediate interest to most users as they are not directly required for (all but the most specialized) scientific data analysis tasks. BCFs contain the lowest level calibration datasets potentially required by downstream software, and can be considered the `atomic units' of the instrument calibration. CPFs contain `convolutions' of the information stored within BCFs customized for a specific analysis task and/or scientific observation.

2.1  BCF & CPF Datasets

All BCF & CPF calibration files are organized using the scheme
/caldb/data/mission/inst
where mission & inst are the OGIP-standard names for the mission and instrument. For internal management purposes, a further division into inst/bcf and inst/cpf sub-directories is made in most cases. Miscellaneous, non-instrument specific calibration datasets, including general spacecraft housekeeping information etc, can be found in the /caldb/data/mission/mis sub-directory.
Besides the calibration datasets themselves, each sub-directory of /caldb/data/mission contains:

2.1.1  Symbolic Links to Other Areas

As mentioned in Section 1.3, there are 2 routes via which ftp users can access BCF & CPF calibration datasets: Both routes access the same physical files, and the two access routes are connected via a symbolic link calib_data in the /mission directory to the /caldb/data/mission directory3
Thus ftp users in (for example) /rosat who change directory to calib_data (ftp> cd calib_data) will actually set their current working directory to /caldb/data/rosat directory. A message to this effect will appear on their screen, along with instructions that at any time they may return to the main /rosat ftp area by changing directory to rosat (ftp> cd /rosat).

2.2  PCFs

Due the the large data volume, for historical reasons the PCFs from all missions are given their own parallel directory tree:
/caldb/data/pcf/mission
where mission in the standard OGIP name for the mission. Each directory is further divided into the following sub-directories:

2.2.1  Symbolic Links to Other Areas

The calibration datasets resulting from the observation of astronomical sources (eg the Crab) are available in the /mission/data/inst directory tree (see Section 1.3). Thus cal_obs will be a symbolic link to this directory. It should be noted that NO corresponding symbolic link from the /mission/data/inst area back to /caldb/data/pcf/mission/inst will be provided. Users wishing to return will therefore be required to use the full path when changing directory back.

2.3  Non-FITS files

At the present time the HEASARC CALDB does contain a number of calibration datasets in formats other than FITS. In some cases, FITS versions of these datasets (using a standard OGIP file format) also exist within the caldb. Users should note that the permanent existence, quality etc of these non-FITS files is in NO WAY guaranteed. In most cases such files represent calibration datasets in an old format. Thus such files are stored in an old sub-directory of the .../inst/bcf or .../inst/cpf directories. The HEASARC tries to provide ASCII README files to explain the formats etc of the datasets, and giving the name and location of the corresponding FITS versions of the file, when available.

3  CALIBRATION SOFTWARE

3.1  CALTOOLS & CALLIB

Calibration-related software is publicly available from the caltools package distributed with the HEASARC ftools analysis software. All routines are written in ANSI FORTRAN or C, and are able to be executed both as standalone tasks on all OGIP-supported platforms/operating systems, or from within the IRAF environment. A list of all currently available and planned caltools tasks is given on
http://heasarc.gsfc.nasa.gov/lheasoft/ftools/caldb/caltools.html.
Users may also be interested in the callib subroutine library (in /caldb/software/callib) containing commonly used utilities, and in particular, routines for reading and writing calibration datasets adhering to OGIP-standard file formats. A detailed listing of the contents of callib can be found in "Summary of CALLIB Routines", CAL/SW/93-005.
Details on the installation and maintenance of the HEASARC caldb software collection at remote nodes can be found in "How to Manage a Calibration Database", CAL/GEN/92-015.

3.1.1  Symbolic Links to Other Areas

The /caldb/software/caltools file is a symbolic link to the /software/ftools/caltools area within the main HEASARC software collection. Similarly /caldb/software/callib is a symbolic link to the /software/ftools/callib area.
It should be noted that NO corresponding symbolic links from these directories back to /caldb/software are provided. Users wishing to return will therefore be required to use the full path of the desired directory when changing back.

4  CALIBRATION DOCUMENTATION

The /caldb/docs directory is sub-divided into In most cases both the LaTeX source code, Postscript & HTML versions of each document are available.
The best starting point for finding calibration-related documentation is via the World-wide Web via the URL:
/docs/heasarc/caldb/caldb_doc.html

REFERENCES

Drake, S., & O'Neel, B., 1993. Legacy, 3, 53.
George, I.M., 1992. Legacy, 1, 56. (CAL/GEN/91-001)
George, I.M. & Zellar, R., 1993. In preparation. (CAL/GEN/92-015)
George, I.M., Zellar, R. & Yusaf, R, 1993a. In preparation. (CAL/SW/93-004)
George, I.M., Zellar, R. & Yusaf, R, 1993b. In preparation. (CAL/SW/93-005)
Pence, W., 1992. Legacy, 1, 14.

USEFUL LINKS TO OTHER HTML PAGES

The following useful links are available (in the HTML version of this document):

Footnotes:

1We generally use unix-style directory paths throughout this memo for simplicity.
2This will be the / directory for ftp users, and the /FTP directory for non-ftp users.
3NOTE This symbolic link ONLY works for ftp users. Users not logged onto legacy via ftp unfortunately have to specify the path to the relevant caldb area either using:

File translated from TEX by TTH, version 3.73.
On 30 Mar 2006, 12:04.