Calibration Memo CAL/GEN/92-008

Calibration Index Files

Ian M George

Bill Pence

& Ron S Zellar
Codes 664 & 668,
NASA/GSFC,
Greenbelt,
MD 20771



Version: 1995 Mar 01






SUMMARY

The role and detailed format of Calibration Index Files (CIFs) within the Calibration Database (caldb) are discussed. The associated s/w tasks crcif & udcif are also briefly described, as are a number of other tasks and FORTRAN calibration subroutine library (callib) tasks involved in the access of CIFs. References are provided to documentation for the installation and maintenance of Calibration Database (caldb) s/w & data at remote sites.

Intended audience: primarily authors of downstream s/w, general users of OGIP calibration data at remote sites.

LOG OF SIGNIFICANT CHANGES

Release Sections Changed Brief Notes
Date
1993 Feb 22 Original Version
1995 Mar 01 All Made compatible with LaTeX2HTML software
2004 Apr 01 All Made compatible with tth

1  INTRODUCTION

In line with IAU and NASA policy, all files within the OGIP will make use of the Flexible Image Transoprt System (FITS; eg. see Wells et al. 1981, Griesen & Harten 1981). The files will make use of (and conform to) all the recent enhancements of the original FITS formats. Specifically, wide use will be made of 'extensions' (Grosbol et al. 1988), 'ASCII tables' (Harten et al. 1988), and, in particular, 'binary tables' (BINTABLE; Cotton & Tody 1992) FITS enhancements. The files within the OGIP Calibration Database (caldb) provide no exception to this.

Due to various logistic reasons, and as an aide to clarity, it is proposed that the calibration data is stored in a relatively large number of small files within the caldb. Broadly speaking, each file will be dedicated to a single aspect of calibration, and consist of a single FITS extension containing a single calibration dataset. In some cases, however, a few closely related datasets may be combined in a single FITS extension in order to save disk space, and for clarity. In the event of a calibration update, the new calibration information will be written into a new file, which (wherever possible) employs the same format corresponding old calibration file. The old file, however, will of course be retained within the caldb and accessible to users and downstream s/w.

The advantages of such a scheme are:

whilst the main disadvantage is: Strict configuration control and database management is therefore clearly required. The FITS header of each calibration file will contain all neccessary details concerning the contents, orgin, time-tagging for use, and (to a limited extent) information on the use of the data within. Such information will also be given in greater detail in relevant documentation (eg. the 'OGIP Calibration Guide' for each instrument). However, standard information concerning the name, location, format and brief summary of contents of each calibration dataset within each file within the OGIP caldb will also be stored in "Calibration Index Files". It is intended that these files play a crucial role in this file-housekeeping both within the OGIP and at a remote user's home institution. They therefore form the subject of this memo.

In Section 2 the design requirements of CIFs are described, along with a brief discussion of their role within the OGIP Calibration Database (caldb) and their interaction with downstream s/w. A detailed description of the format of a CIF file is given in Section 3. Then, in Section 4 we briefly review the requirements (mandatory keywords) imposed on all calibration files to be included within the OGIP caldb. In Section 5 a number of associated s/w tasks and routines are listed. Finally in Section 6, references are provided to other documentation which detail the creation and maintenance of CIFs, both within the OGIP caldb, and at remote sites.

2  THE PURPOSE OF CALIBRATION INDEX FILES

A primary goal of the HEASARC is to make all s/w as mission/detector independent as possible. To this end, i/p filenames etc. are not hardwired into the code. In the case of calibration data, this policy obviously gives users & s/w greater flexiblity in choosing which calibration data they wish to use. Of course in most cases, users will desire to use the standard (best) calibration data, and this will be the default case. It will be the responsibility of the user to ensure any non-default calibration data they use makes sense.

Calibration Index Files (CIF) have been designed to fulfill the following functions:

  1. To enable down-stream s/w to locate required calibration information.

  2. To enable the HEASARC to keep track of the numerous calibration files within the archive.

  3. To provide users with a concise summary of the available calibration information. (A copy of the information within the CIF will also be provided on-line at the HEASARC, accessible through the XOBSERVER account, enabling users to see (BROWSE) what's available.)

It is anticipated that the majority of GOF/HEASARC will wish to perform the bulk of their analysis on their own machines. These users will thus need to copy some/all the OGIP Calibration Database (caldb) to their own machine, along with any required HEASARC software. It is therefore intended that CIFs also perform the above functions for users at their home institution.

2.1  Design Requirements & Constraints

The CIFs were designed with the following requirements in mind:

  1. The format of the file must be machine independent, and user-friendly (ie. FITS).

  2. The format must contain all the information necessary for s/w to locate the required file and calibration dataset stored within.

  3. The CIFs should be able to be stored anywhere on disk (so as to reduce the restrictions imposed on remote users wishing their own copy of the OGIP Calibration Database (caldb) and associated software).

In order to simplfy s/w and user access, the following constraints have been imposed:

  1. A single CIF must contain all the information concerning the calibration files associated with a given instrument (though several instruments may share the same CIF if it is so desired).
    (This is equivalent to saying that all the calibration files associated with a given instrument must reside under a single directory tree, though several instruments can share the same tree).

2.2  Overview of Access to & Role of CIFs

2.2.1  System Concerns

In order to reduce disk-space and maintenance overheads, it is strongly recommended that remote users on clustered machines share a single local caldb, and that a single person be responsible for it's maintenance.

  1. It is strongly recommended (especially on clustered machines) that a system-wide symbolic link (under unix/ultrix) or logical (under VMS) be defined pointing to the physical location of the local caldb sub-directories. Within the OGIP caldb, the 'logical' caldb is used for this purpose. Remote users are free to choose their own 'logical', but may wish to adopt the same convention as the HEASARC thus

    (This is therefore an identical setup to that recommended to define the XANADU 'logical' at remote nodes)

  2. The disk location and name of the various CIFs is stored using environment variables defined in a site-specific (calibration) initialization file (cinitu.csh under the unix C shell). Examples of the contents of this file relevant to CIFs are given in Table 1 (in which all CIFs have the same name, caldb.indx, for calrity). This file must be executed prior to using any calibration s/w (and will be sourced by the appropriate go_caldb file). It is the responsibility of the remote user with a local caldb to ensure that the appropriate pointers within this file are defined correctly. This is discussed in more detail in CAL/GEN/92-015 (George & Zellar 1993a).

Table 1: Example of contents of cinitu.vms and cinitu.csh files required for CIFs
under VMS (cinitu.vms)
........
ASS caldb:[data.astrod.gis]caldb.indx asdgiscif:
ASS caldb:[data.astrod.sis]caldb.indx asdsiscif:
ASS caldb:[data.astrod.xrt]caldb.indx asdxrtcif:
ASS caldb:[data.rosat.pspc]caldb.indx rospspccif:
ASS caldb:[data.rosat.hri]caldb.indx roshricif:
etc.
........
$
under the unix/ultrix C shell (cinitu.csh)
........
setenv asdgiscif /caldb/data/astrod/gis/caldb.indx
setenv asdsiscif /caldb/data/astrod/sis/caldb.indx
setenv asdxrtcif /caldb/data/astrod/xrt/caldb.indx
setenv rospspccif /caldb/data/rosat/pspc/caldb.indx
setenv roshricif /caldb/data/rosat/hri/caldb.indx
etc.
........

2.2.2  User Access

The CIFs will be in FITS format and hence readily accessible to users (via FTOOLS or a similar general FITS reader) wishing to inspect all the calibration datasets currently available.

Obviously, once the location and name of the desired calibration dataset has been obtained, users are able to use other FTOOLS etc. to inspect, extract & manipulate the data file themselves.

2.2.3  Software Access to Calibration Datasets and the Role of CIFs

Software access to calibration datasets via CIFs is as follows:

  1. A CALTOOLS task quzcif (or callib subroutine library routine qzcif, see Section 5.3.1), is called from the main s/w task. This finds and opens the required CIF, searchs it for the requested (passed) entry, and if found, returns the name & disk location of the file containing the required calibration information. A more advanced mode of operation is also possible in which the user can be prompted for which version of a given calibration dataset they wish to use - see Section 5.3.1.

  2. The main s/w task then finds and opens this calibration file, and hence is able to access the extension containing the required calibration dataset.

In most cases, higher level tasks within the OGIP calibration FORTRAN subroutine library (callib) are available to perform the above 2 steps (see Section 5.3.2).

3  CALIBRATION INDEX FILE FORMAT

The Calibration Index Files consist of a FITS file with a null primary array and a single BINTABLE extension. Each row within the extension refers to a single calibration dataset within a single extension in a single calibration file. The values inserted into the various columns of the BINTABLE originate either from the calibration file or are supplied by the installation s/w (crcif, udcif) as noted below. As stated above, when taken together each row of the CIF contains all the information for downstream s/w to identify, locate a required calibration dataset.

3.1  The INDEX Extension

3.1.1  Extension Header

Besides the standard FITS keywords, the header contains the following (mandatory) keywords/values:

3.1.2  Data (Table) Format

The BINTABLE of the CIF consists of the following columns:

  1. Telescope, a 10-byte CHARACTER string specifying the telescope/mission to which the calibration data applies (see CAL/GEN/92-011, George, Zellar & Pence 1993, for a summary of allowed values, available on-line as pdf and html versions).
    The FITS column name is TELESCOP, and the value obtained directly from the TELESCOP keyword in the calibration file.

  2. Detector, a 10-byte CHARACTER string specifying the detector/instrument to which the calibration data applies (see CAL/GEN/92-011, George, Zellar & Pence 1993, for a summary of allowed values, available on-line as pdf and html versions).
    The FITS column name is INSTRUME, and the value obtained directly from the INSTRUME keyword in the calibration file.

  3. Detnam, a 20-byte CHARACTER string further specifying the specific detector to which the calibration refers (ie. when the value of Detector is insufficient). CAL/GEN/92-011 (George, Zellar & Pence 1993, available on-line as pdf and html versions) gives a list of instruments for which this keyword is often required along with a summary of allowed values.
    The FITS column name is DETNAM, and the value obtained directly from the DETNAM keyword in the calibration file (if present).

  4. Filter, a 10-byte CHARACTER string specifying the filter in use. This column will contain the string 'NONE' in the case of instruments without a moveable filter or for calibration datasets for which filter information is not relevant (see CAL/GEN/92-011, George, Zellar & Pence 1993, for a summary of allowed values, available on-line as and html versions).
    The FITS column name is FILTER, and the value obtained directly from the FILTER keyword in the calibration file (if present).

  5. Log_Link, a 20-byte CHARACTER string specifying the system-wide symbolic link (under unix/ultrix) or system-wide logical (under VMS) defining the physical location (disk) of the calibration sub-directories within which the file resides (ie. CALDB, or whatever, as defined in Section 2.2.1).
    The FITS column name is CAL_DEV, and the appropriate value inserted by crcif, udcif etc.

  6. Directory, a 70-byte CHARACTER string specifying the sub-directory location of the calibration file beyond that given by Log_Link. The string will conform to OGIP standards for such directory paths whereby the sub-directory names are specified by a slash ("/"). This is true whatever operating system is in use on either the disk on which the calibration data files resides, or (if different) the disk where the CIF resides.
    The FITS column name is CAL_DIR, and the appropriate value inserted by crcif, udcif etc.

  7. File, a 40-byte CHARACTER string specifying the name of the calibration data file.
    The FITS column name is CAL_FILE, and the appropriate value inserted by crcif, udcif etc.

  8. Calclass, a 3-byte CHARACTER string specifying the OGIP class of calibration file (ie. PCF, BCF, CPF etc. - see CAL/GEN/92-011, George, Zellar & Pence 1993, for a summary of all allowed values, available on-line as and html versions).
    The FITS column name is CAL_CLAS, and the value obtained directly from the CCLSxxxx keyword in the calibration file.
    This presence of this keyword also flags that the calibration dataset is to be included within the CIF.

  9. Caltype, a 4-byte CHARACTER string specifying whether the calibration data consists of real data, or 'virtual' data (ie. a taskname and associated parameter inputs - see CAL/GEN/92-013, George, Zellar & White 1993 for further information).
    The FITS column name is CAL_DTYP, and the value obtained directly from the CDTPxxxx keyword in the calibration file.

  10. Calcode, a 20-byte CHARACTER string giving the OGIP codename describing the calibration dataset.
    The FITS column name is CAL_CNAM, and the value obtained directly from the CCNMxxxx keyword in the calibration file (see CAL/GEN/92-011, George, Zellar & Pence 1993, for a summary of allowed values, available on-line as and html versions).

  11. CALBound, an array of 70-byte CHARACTER strings giving the limitations/boundaries (if any) of the calibration dataset.
    The FITS column name is CAL_CBD, and the values obtained directly from the CBDnxxxx keyword in the calibration file (see CAL/GEN/92-011, George, Zellar & Pence 1993, for a detailed discussion and summary of allowed values, available on-line as pdf and html versions).

  12. Extno, a 2-byte INTEGER giving the extension number of the EXTname extension within File. The FITS column name is CAL_XNO, and the appropriate value inserted by crcif, udcif etc.

  13. Valstart  D, a 8-byte CHARACTER string giving the UTC date (in the format dd/mm/yy) when these calibration data should first be used.
    The FITS column name is CAL_VSD, and the value obtained directly from the CVSDxxxx keyword in the calibration file.

  14. Valstart  T, a 8-byte CHARACTER string giving the UTC time (in the format hh:mm:ss) on day Valstart  D when these calibration data should first be used.
    The FITS column name is CAL_VST, and the value obtained directly from the CVSTxxxx keyword in the calibration file.

  15. Ref T, an 8-byte REAL giving the values of ValstartD and ValstartT converted to modified Julian date for the convenience of access s/w.
    The FITS column name is REF_TIME, and the appropriate value inserted by crcif, udcif etc.

  16. Qual, a 2-byte INTEGER specifying the nominal quality of the data.
    1. Qual=0 if the calibration data quality is considered "good"

    2. Qual ≠ 0 if the calibration data is considered "bad", with the option of the integer value specifying the origin of the flag:
      • Qual=1; data considered ``bad" by h/w team

      • Qual=2; data considered ``dubious" by h/w team

      • Qual=3; data ``bad or dubious" by association

      • Qual=4; spare

      • Qual=5; data considered ``bad" by GOF/HEASARC/User

      • Qual=−1; reason for ``bad" flag unknown

      • Qual=−9; dataset no longer exists (file removed from caldb)

    The FITS column name is CAL_QUAL, and the appropriate value inserted by crcif, udcif etc. either via the optional cif.qual file (see CAL/GEN/92-015, George & Zellar 1993a) or via a prompt to the user. This column can also be edited by EDTCIF (Section 5.3.1).

  17. OGIP Date, a 8-byte character string giving the date (in the format dd/mm/yy) on which this file was installed in the OGIP calibration database.
    The FITS column name is CAL_DATE, and the appropriate value inserted by crcif, udcif etc.

  18. Desc, a 70-byte CHARACTER string giving a brief description the the data stored within this extension of File.
    The FITS column name is CAL_DESC, and the value obtained directly from the CDESxxxx keyword in the calibration file. This column can also be edited by EDTCIF (Section 5.3.1).

These are summarized in Table 2.

Table 2: Summary of the format for CIFs (CIFVERSN = 1992a)

Extension
to caldb.indx
Name: CIF
Description: Calibration Index File.
Format: BINTABLE
column
1 23 4 5 6 7 8 9
contents
Telescope Instr.Detector Filter Environ.Directory Name of OGIP ClassOGIP
or Mission Name Name Name Variable Calibrationof Cal.Caldb
Name to caldb File Dataset Datatype
Telescope Detector Detnam Filter Log_Link Directory File Calclass Caltype
format of each column
10-byte 10-byte 20-byte 10-byte 20-byte 70-byte 40-byte 3-byte 4-byte
char. char. char. char. char. char. char. char. char.
column name
TELESCOP INSTRUME DETNAM FILTER CAL_DEV CAL_DIR CAL_FILE CAL_CLAS CAL_DTYP
column (cont)
10 11 12 13 14 15 16 17 18
contents
OGIP Dataset Extension Validity Start Quality Install. Descript.
dataset Limits Number Date Time Ref Time Flag Date String
Calcode Calbound Extno ValstartD ValstartT Ref T Qual Caldb Date Desc
format of each column
20-byte 70-byte 2-byte 8-byte 8-byte 8-byte 2-byte 8-byte 70-byte
char. char. integer char. char. real integer char. char.
array
column name
CAL_CNAM CAL_CBD CAL_XNO CAL_VSD CAL_VST REF_TIMECAL_QUAL CAL_DATE CAL_DESC

4  CALIBRATION FILE REQUIREMENTS

As should be apparent from Section 3.1.2, the following keywords are mandatory in the header of the FITS extension containing all calibration datasets:

where xxxx is a number of the form 0001, 0002, 0003 etc. These keywords are further described, along with their allowed values in CAL/GEN/92-011 (George, Zellar & Pence 1993, available on-line as and html versions).

It is the responsiblity of those supplying the files for inclusion in the CALDB to ensure that all other relevant keywords are present and correct, and that sufficient COMMENT and/or HISTORY keywords are supplied to fully identify and describe the dataset. It is also the responsibility of suppliers of calibration datasets to provide all necessary documentation concerning the origin, description, use and limitations of each dataset to the manager of the HEASARC Calibration Database (caldb). Send e-mail to caldbhelp@olegacy.gsfc.nasa.gov for more information.

5  ASSOCIATED SOFTWARE TASKS & ROUTINES

All s/w which creates or accesses CIFs will:

5.1  Creation & Updates to CIFs

The following tasks involved with the creation and updating of CIFs are available to the general community:

  1. crcif (`create CIF'). This routine searches for FITS calibration files containing extensions satisfying standard criteria (see CAL/GEN/92-015, George & Zellar 1993a) within a user-defined directory structure. The required keywords (Section 4) from each extension found are then extracted, and written along with the other necessary keywords (see Section 3.1) into a new CIF in a user-defined location.

  2. udcif (`update CIF'). This routine takes a single calibration file currently in a user-defined directory and checks for extensions satisfying the same standard criteria as crcif. The required keywords (Section 4) from each calibration dataset found are then extracted. Following checks that this dataset is not already included within the CIF, the new keywords corresponding to the new dataset are appended along with the other necessary keywords (see Section 3.1) as a new row to a pre-existing (user-defined) CIF.

Both the above tasks have an optional input file (cif.qual) which supplies information concerning the quality of each calibration dataset to be included in the CIF. These tasks, along with the format and use of cif.qual files, are described in more detail in CAL/GEN/92-015 (George & Zellar 1993a).

5.2  The (limited) Editing of CIFs

There are only two occasions when a user may need to edit a CIF:

Both these tasks can be performed using the CALTOOLS task edtcif, which also provides the facility to view the contents of a CIF (see Section 5.3.1). Users are strongly discouraged from any other editing of CIFs.

Obviously only a very restricted number of users ought to have write priviledge (and hence be able to successfully edit) the primary CIFs of any remote nodes. Individual users who wish to customize their own CIF can, however, copy the system CIF to their own space, make any changes they wish, and reset the appropriate pointer from the cinitu file (Section 2.2.1) prior to running any CALTOOLS software.

5.3  CIF Inquiries

5.3.1  Inquiry Tasks

Since CIFs are in FITS format, a large number of general (publically available) FITS tasks are available to users wishing to manipulate their contents. However, clearly the contents and structure of the original CIF must be left intact if OGIP-supplied calibration s/w is expected to work (but see also Section 5.2).

The following CALTOOLS tasks are of use for CIF inquiries (see CAL/SW/93-004; George, Zellar & Yusaf 1993a, available on-line as pdf and html

versions)

  1. quzcif (`quiz CIF'). This task finds (using the environment variables discussed in Section 2.2.1) and opens the specified CIF. It then searches through this CIF for entries (rows) corresponding to the calibration data required (ie. appropriate for the mission & instrument of interest, and with the desired CCNMxxxx codename).
    If being run in :

  2. edtcif (`edit CIF'). This task provides an interactive display and (limited) editing facility for CIFs. Users are able to select which rows & columns are to be displayed, and display the full contents of any given row. The results can also be written out to an ASCII file if desired. Users (with write priviledges) are also able to change the value of the followings columns (only) of the CIF:

The following FUTILS tasks are also of use for user inquiry of CIFs (see Pence 1992)
  1. FDUMP - writes the contents of any FITS file to either the terminal or an ASCII file.

  2. FSELECT - writes a user-selected subset of the contents of any FITS file to a new FITS file.

5.3.2  Inquiry Routines

The following FORTRAN subroutines available in the calibration subroutine library (callib, see CAL/SW/93-005, George, Zellar & Yusaf 1993b, available online in pdf and html versions) are provided to aide s/w inquiry of CIFs:

  1. qzcif, the primary FORTRAN inquiry subroutine for CIFs. This routine is at the heart of the CALTOOLS task quzcif described in Section 5.3.1, and hence is able to perfrom the same tasks as described there.

A number of higher-level individual subroutines will/are also provided within callib which given the mission & instrument name, observation etc., finds (using qzcif), reads & returns the relevant calibration dataset to the main program. An example is the subroutine gt_det_eff which returns the detector efficiency as a function of energy for any given detector1. A more detailed desciption of all such members of callib is given in CAL/SW/93-005 (George, Zellar & Yusaf 1993b, available online in pdf and html versions).

6  INSTALLATION & USE AT REMOTE SITES

Documentation describing the installation, maintenance and use of CIFs:

As stated above, in order to reduce disk-space and maintenance requirements, it is strongly recommended that remote users on clustered machines share a single local caldb, and that a single person be responsible for its maintenance.

ACKNOWLEDGMENTS

We thank the numerous people, both inside and outside the OGIP, who have contributed ideas and suggestions.

REFERENCES

Blackburn, K., 1993. In preparation.

Cotton, W.D. & Tody, D., 1992. In preparation.

George, I.M., 1992. Legacy, 1, 56, CAL/GEN/91-001
(available online in pdf and html versions).

George, I.M. & Zellar, R., 1993a. In preparation. (CAL/GEN/92-015)

George, I.M., Zellar, R., & Pence, W., 1993. OGIP Calibration Memo CAL/GEN/92-011
(available online in pdf and html versions).

George, I.M., Zellar, R., & Yusaf, R., 1993a. OGIP Calibration Memo CAL/SW/93-004
(available online in pdf and html versions).

George, I.M., Zellar, R., & Yusaf, R., 1993b. OGIP Calibration Memo CAL/SW/93-005
(available online in pdf and html versions).

Griesen, E.W. & Harten, R.H., 1981. Astron. Astrophys. Suppl., 44, 371.

Grosbol, P., Harten, R.H., Greisen, E.W. & Wells, D.C., 1988. Astron. Astrophys. Suppl., 73, 359.

Harten, R.H., Grosbol, P., Griesen, E.W. & Wells, D.C., 1988. Astron. Astrophys. Suppl., 73, 365.

Mukai, K., 1992. OGIP Memo OGIP/92-010.

Pence, W., 1992. Legacy, 1, 14..

Wells, D.C., Griesen, E.W. & Harten, R.H., 1981, Astron. Astrophys. Suppl., 44, 363.

Zellar, R. & George, I.M., 1992. In preparation. (CAL/GEN/92-014).

USEFUL LINKS TO OTHER HTML PAGES

The following useful links are available (in the HTML version of this document only):


Footnotes:

1Clearly, such routines require that the relevant OGIP caldb calibration files exist within the local caldb. If this is not the case, then a descriptive error flag is returned to the main program.


File translated from TEX by TTH, version 3.13.
On 4 May 2004, 15:00.