OGIP Calibration Memo CAL/GEN/94-002

OGIP Calibration Memo CAL/GEN/94-002

The Calibration Database User's Guide

Ian M George & Lorraine Breedon
Mail Codes 660.2 & 664,
NASA/GSFC,
Greenbelt, MD20771

Version: 1996 Sep 30

SUMMARY

This document describes how a user should set up their account in order to use a local copy of the HEASARC calibration database.

Forward

This document is intended to provide a general User's Guide to their local copy of the HEASARC's calibration database system. Every attempt has been made to present the information in a clear, and (moderately) concise manner, and copious use of cross-referencing is made, although reference to other Calibration Memos etc is occasionally required.

All questions, comments and suggestions should be directed to:
caldbhelp@athena.gsfc.nasa.gov
or to a member of Team Caldb listed at the URL:
http://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_team.html.

A printable version of this document can be obtained at:
http://heasarc.gsfc.nasa.gov/FTP/caldb/docs/manuals/user/users_guide.pdf.

LOG OF SIGNIFICANT CHANGES

Release	Sections Changed	Brief Notes
Date

1994 Jan 21		First Public Release
1995 Jul 12	All	Made compatible with `LaTeX2HTML` software
1995 Oct 13	All	Updated and expanded
1996 Sep 30	All	Clarifies in the line of user's questions
2004 Apr 01	All	Made compatible with tth
2012 Aug 08	All	minor typos/corrections

1 INTRODUCTION
    1.1 Is a Local Caldb Available to Me?
    1.2 Do I really need Access to the Caldb
2 STANDARD SET-UP FOR YOUR ACCOUNT
    2.1 Setting Up the Caldb Environment
        2.1.1 On unix/ultrix/osf systems
    2.2 Check the CALDB environment
    2.3 Mounting the CALDB Directories
    2.4 Final mission/instrument specific checks
3 CUSTOMIZING YOUR CALDB SET-UP
    3.1 When to Customize
    3.2 How to Customize
4 COMMON PROBLEMS & FAQs
    4.1 I don't understand anything
    4.2 Do I have FTOOLS?
    4.3 Why doesn't the FTOOLS installation set up a local Caldb ?
5 IF ALL ELSE FAILS ...

1 INTRODUCTION

The HEASARC's calibration database (Caldb) system is a flexible, unix-based file system which stores and indexes FITS format calibration datasets. Essentially, the Caldb system is a relatively simple relational-database in which calibration datasets are linked to software via Calibration Index Files (CIFs). CIFs contain information for every dataset in the local Caldb and are read by the Caldb access software (the general user should not need to know any more, but if you're really keen and want further information on CIFs, see CAL/GEN/92-008).

Since a local Caldb can become very large (and there are some maintenance overheads), the HEASARC recommends that a site maintain only one local copy of the Calibration Database on a central file system, that this system is administered by a single 'local Caldb manager', and that the local Caldb be made available to multiple users at that site. Further information regarding the set-up and maintenance of a local Caldb is available in the OGIP Calibration Memos CAL/GEN/94-004 and CAL/GEN/92-015.

The Caldb access software is distributed through the FTOOLS software package. If a local caldb already exists at your site, most likely the FTOOLS software is also available to you. Again, the reader should verify that indeed FTOOLS are available at their institution and determine its location (see Section 4.2). It should be stressed that a local Caldb is NOT automatically set up by the FTOOLS installation (see Section 4.3 for more information).

1.1 Is a Local Caldb Available to Me?

The task caldbinfo distributed with the FTOOLS software package (starting with version 3.5)provides a simple test as to whether a local Caldb has been set-up and is accessible to you.

At the operating system prompt, type:
caldbinfo infomode=BASIC chatter=30

if
NO
the task caldbinfo did not run
- then FTOOLS are probably not available to you. Go to Section 4.2.
if
YES
the task caldbinfo did run
- but if
  NO
  your local Caldb does not exist (or is inaccessible to you), and a heap of moderately self-explanatory error messages are written to STDOUT, then go to Section 2.1.
- otherwise if
  YES
  and a number of informative messages are written to STDOUT, then it appears that you already have a local Caldb set-up and accessible to you. Go to Section 2.

caldbinfo has additional functionality which may help should you experience problems at a latter time. Check-out the help file (type fhelp caldbinfo at the system prompt) for more information.

1.2 Do I really need Access to the Caldb

Caldb access is helpful for calibration and analysis of many of the high-energy missions archived at the HEASARC (notably CHANDRA, ROSAT, Swift, Suzaku, RXTE, and other missions listed on the Caldb "Supported Missions" web page.

Several FTOOLS tasks contain code which enables them to access a Caldb. Generally these tasks will contain a parameter for the calibration file(s) to be used. Entering the string caldb will inform the task that the Caldb is to be asked to supply the name & location of the appropriate calibration files. If you do not have access to a local Caldb, then clearly you will have to find the appropriate file (which can be far from trivial) and supply the name of that file via the appropriate XPI parameter.

It's important to note that you don't necessarily need to download or maintain a local version of the Caldb. Most FTOOLS which support Caldb access also support Remote Access to the HEASARC Caldb. For more information on remotely accessing the HEASARC Caldb, please see the Remote Access Webpage, http://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_remote_access.html.

In short Caldb access is not required, but it can make your life easier. It is relatively trivial to set up Caldb access (either by accessing a local copy of the Caldb, or accessing the HEASARC's caldb over the internet) and will make the running of many tasks much less painful.

2 STANDARD SET-UP FOR YOUR ACCOUNT

There are two basic requirements for a user to be able to use their local Caldb:

that they define a number of environment variables (or logicals) which allow the Caldb software to find the Calibration Index Files (CIFs) and hence the calibration datasets themselves. The set-up of this environment variables (or logicals) is achieved by `sourcing' a single file as descibed in Section 2.1.
that they have read access to the disk, file system etc containing the data files (Section 2.3).

2.1 Setting Up the Caldb Environment

The Caldb environment is a set of user-defined environment variables which point to Calibration Index Files (CIFs). CIFs contain calibration information for every dataset in the Caldb and are read by the Caldb access software (for further information on CIFs, see CAL/GEN/92-008). The Caldb environment can be set up by sourcing the appropriate caldbinit file as explained below.

At the present time, the caldbinit file defines the following environment variables:

CALDB - the path to the top-level caldb directory on your system
CALDBCONFIG - the path and name of the CALDB-configuration file, which gives the locations of CIFs for each supported mission.
CALDBALIAS - the path and name of the CALDB instrument-alias file. This (FITS) file contains information used internally by the CALDB software in cases where an alias exists for one or more instruments. This is used since, in some Caldb files, certain INSTRUME keyword values imply that the dataset is valid for multiple instruments. When updating a Calibration Index file with a new calibration dataset using the FTOOL udcif, the CALDBALIAS file is checked, and, if such a value is found for the dataset being indexed, then the INSTRUME value is translated into its alias values and a separate entry is made in the CIF for each. For example, for ASCA, INSTRUME = 'GIS' will cause UDCIF to create an entry for GIS2 and GIS3 in the Calibration Index File.

These files should have been edited appropriately by your local Caldb manager, and only users wishing to customize a personal caldb will have to worry about its contents (see Section 3).

2.1.1 On unix/ultrix/osf systems

Linux or Unix:
If you are using a machine running unix/linux,and want to use your local Caldb directly from your operating system, add the following line to your .login file:
source path/caldbinit.csh
(if you are running c-shell or t-shell), or
source path/caldbinit.sh
(if you are running the Bourne shell), where path is the path to the directory in which the caldbinit file is kept. Generally it is recommended that this path is /caldb/software/tools, however your local Caldb manager may have chosen to put the caldbinit.unix file somewhere else. If you are unable to locate the appropriate caldbinit file, contact your local Caldb manager.

After this line has been added to your .login file, source that file (source $HOME\.login) or logout & login again.

2.2 Check the CALDB environment

One can verify that the CALDB environment has successfully been set-up by repeating the test using the FTOOLS software package caldbinfo outlined in Section 1.1, namely at the operating system prompt, type:
caldbinfo infomode=BASIC chatter=30

if
NO
the task caldbinfo did not run
- then FTOOLS are probably not available to you. Go to Section 4.2.
if
YES
the task caldbinfo did run
- but if
  NO
  your local Caldb environment is incorrectly set up. Go back and repeat the instructions given in Section 2.1. If it continues to fail, contact your local Caldb manager for help.
- otherwise if
  YES
  and a number of informational messages are written to STDOUT, then it appears that your local Caldb environment is successfully set-up. Go to Section 2.3.

2.3 Mounting the CALDB Directories

Check that you are able to see your local Caldb directories & files from your account. In Section 1.1 (and/or Section 2.2), you were requested to run the FTOOL caldbinfo as follows:
caldbinfo infomode=BASIC chatter=30
amongst the output should be the lines:
...... environ-var/logical CALDB defined
......... CALDB path = path
The location path is the path to the top-level of your local Caldb.

Check that this file-system is visible to you.

if
NO
path appears not to be visible to your account
- contact your local system administrator (who should either auto-mount the relevant disk on your machine, or tell you from which machines the relevant disk will be visible - cross-check with your local Caldb manager if you have difficulties)
if
YES
you can see the top-level of your local CALDB path
- Go to Section 2.1

2.4 Final mission/instrument specific checks

Finally, do should check that your local Caldb is actually set up for the specific mission/instrument in which you're interested. Total flexibility is given to local Caldb Managers regarding which mission/instruments they include within their local Caldb (primarily for disk-space reasons).

Again, using the FTOOLS software package caldbinfo, but with the infomode parameter set to infomode=INST, you can check your local Caldb is set up for your favourite mission/instrument. At the operating system prompt, type:
caldbinfo infomode=INST chatter=30

if
NO
the task caldbinfo did not run
- then FTOOLS are probably not available to you. Go to Section 4.2.
if
YES
the task caldbinfo did run
- then you will be prompted for the mission & instrument in which you're insterested. For example choosing the mission ASCA and instrument SIS0 will result in o/p including lines to the effect of:
  - either
    ERROR - caldb_info 1.1.1: CALDB NOT correctly configured for the SIS0 instrument onboard ASCA
    If this is the case, then contact your local Caldb manager if your local Caldb can be expanded to include the offending mission/instrument(s).
  - or
    ...... CALDB is configured for the SIS0 instrument onboard ASCA
    ......... Cal Index File: path1/file
    ......... Data directory: path2
    If this is the case, then you should be set up. Go to:
    - Section 3 - if you want to customize your set-up
    - Section - if you want further information on the Caldb access software
    - Section 4 - if experience any problems accessing your local caldb
    - Section 5 - if you get to your wits' end and want to know how to complain

3 CUSTOMIZING YOUR CALDB SET-UP

The Caldb system is moderately flexible such that some degree of customization of a user's set-up is available. However such customization is only recommended for advanced users who fully understand the implications. It is intended that the standard set-up described in Section 2 will satisfy the needs of general users.

3.1 When to Customize

Essentially, the customization involves construction a personal copy of one or more CIFs in order to be able to add to or replace standard calibration datasets with personal copies. Following the customization, these CIFs will be used instead of the standard CIFs available as part of the local Caldb. This is achieved by re-defining the environment variable (or logicals) by which the caldb software locates the relevant files such as to point to some/all files not in a user's official local Caldb. Users effectively become local Caldb managers of a personal Caldb (though they may share some - but not all - datasets with their official local Caldb).

Such customization is recommended only for users who

have a detailed knowledge of the calibration datasets, and wish to investigate the effect on their scientific results of using personal calibration (or developmental calibration datasets not yet released to the community)
wish to use standard calibration datasets not available via their local Caldb (if, for instance, their local Caldb does not include files for a given mission/instrument).

All other users are strongly advised to use the standard set-up described in Section 2.

3.2 How to Customize

As stated in Section 2:

the path to the top of the local caldb directory tree is defined by the CALDB environment variable (logical)
all the environment variables (or logicals) pointing to the CIFs etc are defined by sourcing a file defined by the CALDBCONFIG environment variable (logical).

Under the standard set-up, both CALDB and CALDBCONFIG are defined in the caldbinit file (see Section 2.1).

Customization requires defining the Caldb environment variable CALDB to the appropriate top-level directory pointing to the top-level directory containing the desired Caldb data. Also if necessary the CALDBCONFIG file may need to have an entry added to point to the new Calibration Index File. Consult the HEASARC Caldb Manager for more details or assistance.

4 COMMON PROBLEMS & FAQs

4.1 I don't understand anything

No problem, we'll help you. Don't be shy, just contact us via one of the routes given in Section 5.

4.2 Do I have FTOOLS?

All the Caldb access software is distributed through the FTOOLS software package. Again, the reader should verify that indeed FTOOLS are available at their institution and determine its location. The easiest way to check for FTOOLS is to type the following at the operating system prompt:
fhelp ftools

if
NO
the command is not recognized
- then FTOOLS are not available to you. FTOOLS can be downloaded with the HEASOFT analysis package from http://heasarc.gsfc.nasa.gov/docs/software/lheasoft/download.html.

4.3 Why doesn't the FTOOLS installation set up a local Caldb ?

The FTOOLS installation does NOT automatically include the setting up of a local Caldb. The reasons for this are:

Primarily Disk-space !
The total data holding of the CALDB at NASA/GSFC is many Gbytes. This includes files flagged as both "Good" and "bad" for all the mission/instruments for which the HEASARC has data. It is extremely likely that any given site will only require the "Good" files which constitute only a small fraction of the total data volume. We believe users are best equiped to manage their own disk-space, and anticipate most sites will want to be selective as to what calibration datasets they copy for selected mission/instruments.
The Caldb system allows remote users/sites some flexibility to customize their set-up (see Section 3), allowing then to add personal/locally-produced calibration datasets for both mission/instruments which the HEASARC supports and other mission/instruments.

5 IF ALL ELSE FAILS ...

You can always get help at the HEASARC Feedback page, http://heasarc.gsfc.nasa.gov/cgi-bin/Feedback. Please choose the "Calibration Data Base" mailing list from the pull-down menu.

USEFUL LINKS TO OTHER HTML PAGES

The following useful links are available (in the HTML version of this document only):

File translated from T_EX by T_THgold, version 4.00.
On 10 Aug 2012, 14:42.