OGIP Calibration Memo CAL/GEN/94-002

OGIP Calibration Memo CAL/GEN/94-002

The Calibration Database User's Guide

Ian M George & Lorraine Breedon
Mail Codes 660.2 & 664,
NASA/GSFC,
Greenbelt, MD20771

Version: 1996 Sep 30

SUMMARY

This document describes how a user should set up their account in order to use a local copy of the HEASARC calibration database.

Forward

This document is intended to provide a general User's Guide to their local copy of the HEASARC's calibration database system. Every attempt has been made to present the information in a clear, and (moderately) concise manner, and copious use of cross-referencing is made, although reference to other Calibration Memos etc is occasionally required.

At the current time, this document is very much `live' in the sense that it is being continually updated, and not guarenteed complete.

All questions, comments and suggestions should be directed to:
caldbhelp@athena.gsfc.nasa.gov
or to a member of Team Caldb listed at the URL:
/docs/heasarc/caldb/caldb_team.html.

A postscript file of the latest version of this document can be obtained from the anonymous ftp account on: ftp://legacy.gsfc.nasa.gov/caldb/docs/manuals/user/users_guide.pdf.

LOG OF SIGNIFICANT CHANGES

Release	Sections Changed	Brief Notes
Date

1994 Jan 21		First Public Release
1995 Jul 12	All	Made compatible with `LaTeX2HTML` software
1995 Oct 13	All	Updated and expanded
1996 Sep 30	All	Clarifies in the line of user's questions
2004 Apr 01	All	Made compatible with tth

1 INTRODUCTION
    1.1 Do I have it already ?
    1.2 Do I really need it ?
2 STANDARD SET-UP FOR YOUR ACCOUNT
    2.1 Setting Up the Caldb Environment
        2.1.1 On unix/ultrix/osf systems
        2.1.2 On VMS systems
    2.2 Check the CALDB environment
    2.3 Mounting the CALDB Directories
    2.4 Final mission/instrument specific checks
3 CUSTOMIZING YOUR CALDB SET-UP
    3.1 When to Customize
    3.2 How to Customize
4 CALDB ACCESS TASKS & UTILITIES
5 COMMON PROBLEMS & FAQs
    5.1 I don't understand anything
    5.2 Do I have FTOOLS?
    5.3 How do I get FTOOLS?
    5.4 Why doesn't the FTOOLS installation set up a local Caldb ?
6 IF ALL ELSE FAILS ...

1 INTRODUCTION

The HEASARC's calibration database (Caldb) system is a flexible, system-independent file-system which stores and indexes FITS format calibration datasets. Essentially, the Caldb system is a relatively crude relational-database in which calibration datasets are linked to users and software via Calibration Index Files (CIFs). CIFs contain information for every dataset in the local Caldb and are read by the Caldb access software (the general user should not need to know any more, but if you're really keen and want further information on CIFs, see CAL/GEN/92-008).

Since a local Caldb can become very large (and some there are some maintenance overheads), the HEASARC recommends that a site maintain only one local copy of the Calibration Database on a central file system, that this system is administed by a single 'local Caldb manager', and that the local Caldb be made available to multiple users at that site. The reader should verify that a local Caldb is available at their institution and determine its location. If a local Caldb is NOT available at their site, a user should consider setting one up themselves, and becoming the local Caldb manager. Further information regarding the set-up and maintenance of a local Caldb is available in the OGIP Calibration Memos CAL/GEN/94-004 and CAL/GEN/92-015.

The Caldb access software is distributed through the FTOOLS software package. If a local caldb already exists at your site, most likely the FTOOLS software is also available to you. Again, the reader should verify that indeed FTOOLS are available at their institution and determine its location (see Section 5.2). It should be stressed that a local Caldb is NOT automatically set up by the FTOOLS installation (see Section 5.4 for why not), but most of the necessary components ARE supplied via FTOOLS

1.1 Do I have it already ?

The task caldbinfo is distributed with the FTOOLS software package (starting with version 3.5). Assuming FTOOLS are accessible for your account, then this task provides a simple test as to whether a local Caldb has been set-up and is accessible to you.

At the operating system prompt, type:
caldbinfo infomode=BASIC chatter=30

if
NO
the task caldbinfo did not run
- then FTOOLS are probably not available to you. Go to Section 5.2.
if
YES
the task caldbinfo did run
- but if
  NO
  your local Caldb does not exist (or is inaccessible to you), and a heap of moderately self-explanitory error messages will be written to STDOUT, then go to Section 2.1.
- otherwise if
  YES
  and a number of informational messages are written to STDOUT, then it appears that you already have a local Caldb set-up and accessible to you. Go to Section 2.

caldbinfo has additional functionality which may help should you experience problems at a latter time. Check-out the help file (fhelp caldbinfo) for more information.

1.2 Do I really need it ?

Several FTOOLS tasks contain code which enables them to access a local Caldb. Generally these tasks will contain an XPI parameter for the calibration file(s) to be used. Entering a ßpecial string" (commonly CALDB) will inform the task that the Caldb is to be asked to supply the name & location of the appropriate files. If you do not have access to a local Caldb, then clearly you will have to find the appropriate file (which can be far from trivial) and supply the name of that file via the appropriate XPI parameter.

In short - no you dont have to make your local Caldb accessible. However as discussed below, it should be relatively trivial to set up access (especially if your site already has a well-maintained local Caldb), and will make the running of many tasks much less painful.

2 STANDARD SET-UP FOR YOUR ACCOUNT

There are two basic requirements for a user to be able to use their local Caldb:

that they define a number of environment variables (or logicals) which allow the Caldb software to find the Calibration Index Files (CIFs) and hence the calibration datasets themselves. The set-up of this environment variables (or logicals) is achieved by 'sourcing' a single file as descibed in Section 2.1.
that they can see the disk, file system etc containing the data files (Section 2.3).

2.1 Setting Up the Caldb Environment

The Caldb environment is a set of user-defined environment variables (under unix/ultrix/osf) or logicals (under vms) which point to Calibration Index Files (CIFs). CIFs contain calibration information for every dataset in the local Caldb and are read by the Caldb access software (for further information on CIFs, see CAL/GEN/92-008). The Caldb environment can be set up by sourcing the appropriate caldbinit file as explained below.

At the present time, the caldbinit file defines the following environment variables:

CALDB - the path to the caldb directory structure on your system
CALDBCONFIG - the path and name of the CALDB-configuration file. It is in this file, by default named caldb.config, that all the environment variables (or logicals) pointing to the CIFs are defined.
CALDBALIAS - the path and name of the CALDB instrument-alais file. This (FITS) file contains information used internally by the CALDB software in cases where an alias exists for one or more instruments.

These files should have been edited appropriately by your local Caldb manager, and only users wishing to customize a personal caldb will have to worry about its contents (see Section 3).

2.1.1 On unix/ultrix/osf systems

Host Version:
If you are using a machine running unix/ultrix/osf, and want to use your local Caldb from directly from your operating system, add the following line to your .login file:
source path/caldbinit.unix
where path is the path to the directory in which the caldbinit.unix file is kept. Generally it is recommended that this path is /caldb/software/tools, however your local Caldb manager may have chosen to put the caldbinit.unix file somewhere else. If you are unable to locate the caldbinit.unix file, contact your local Caldb manager.

After this line has been added to your .login file, source that file (source $HOME\.login) or logout & login again.

Under IRAF:
If you are using a machine running IRAF on a unix/ultrix/osf system, and want to use your local Caldb from within the IRAF enironment, add the following line to your loginuser.cl file whilst outside the IRAF environment:
cl < path/caldbinit_iraf.unix
where path is the path to the directory in which the caldbinit_iraf.unix file is kept. Generally it is recommended that this path is /caldb/software/tools, however your local Caldb ,manager may have chosen to put the caldbinit_iraf.unix file somewhere else. If you are unable to locate the caldbinit_iraf.unix file, contact your local Caldb manager.

2.1.2 On VMS systems

Host Version:
If you are using a machine running VMS and want to use your local Caldb from directly from your operating system, add add the following line to your LOGIN.COM file:
@pathCALDBINIT.VMS
where path is the path to the directory in which the CALDBINIT.VMS file is kept. Generally it is recommended that this path is CALDB:[SOFTWARE.TOOLS], however your local Caldb manager may have chosen to put the CALDBINIT.VMS file somewhere else. If you are unable to locate the CALDBINIT.VMS file, contact your local Caldb manager.

After this line has been added to your LOGIN.COM file, source that file (@SYS$LOGIN:LOGIN.COM) or logout & login again.

Under IRAF:
IRAF does not appear to work under a vms system at the current time. Unfortunately your local Caldb will therefore not be available under IRAF on vms systems. However users are reminded that it is not necessary to run (and no advantages are obtained by running) under IRAF. Thus we suggest users simply use the 'Host' version described above.

2.2 Check the CALDB environment

One can verify that the CALDB environment has successfully been set-up by repeating the test using the FTOOLS software package caldbinfo outlined in Section 1.1, namely at the operating system prompt, type:
caldbinfo infomode=BASIC chatter=30

if
NO
the task caldbinfo did not run
- then FTOOLS are probably not available to you. Go to Section 5.2.
if
YES
the task caldbinfo did run
- but if
  NO
  your local Caldb environment is incorrectly set up. Go back and repeat the instructions given in Section 2.1. If it continues to fail, contact your local Caldb manager for help.
- otherwise if
  YES
  and a number of informational messages are written to STDOUT, then it appears that your local Caldb environment is successfully set-up. Go to Section 2.3.

2.3 Mounting the CALDB Directories

Check that you are able to see your local Caldb directories & files from your account. In Section 1.1 (and/or Section 2.2), you were requested to run the FTOOLScaldbinfo as follows:
caldbinfo infomode=BASIC chatter=30
amongst the output should be the lines:
...... environ-var/logical CALDB defined
......... CALDB path = path
The location path is the path to the top-level of your local Caldb.

Check that this file-system is visible to you.

if
NO
path appears not to be visible to your account
- contact your local system administrator (who should either auto-mount the relevant disk on your machine, or tell you from which machines the relevant disk will be visible - cross-check with your local Caldb manager if you have difficulties)
if
YES
you can see the top-level of your local CALDB path
- Go to Section 2.1

2.4 Final mission/instrument specific checks

Finally, do should check that your local Caldb is actually set up for the specific mission/instrument in which you're interested. Total flexibility is given to local Caldb Managers regarding which mission/instruments they include within their local Caldb (primarily for disk-space reasons).

Again, using the FTOOLS software package caldbinfo, but with the infomode parameter set to infomode=INST, you can check your local Caldb is set up for your favourite mission/instrument. At the operating system prompt, type:
caldbinfo infomode=INST chatter=30

if
NO
the task caldbinfo did not run
- then FTOOLS are probably not available to you. Go to Section 5.2.
if
YES
the task caldbinfo did run
- then you will be prompted for the mission & instrument in which you're insterested. For example choosing the mission ASCA and instrument SIS0 will result in o/p including lines to the effect of:
  - either
    ERROR - caldb_info 1.1.1: CALDB NOT correctly configured for the SIS0 instrument onboard ASCA
    If this is the case, then contact your local Caldb manager if your local Caldb can be expanded to include the offending mission/instrument(s).
  - or
    ...... CALDB is configured for the SIS0 instrument onboard ASCA
    ......... Cal Index File: path1/file
    ......... Data directory: path2
    If this is the case, then you should be set up. Go to:
    - Section 3 - if you want to customize your set-up
    - Section 4 - if you want further information on the Caldb access software
    - Section 5 - if experience any problems accessing your local caldb
    - Section 6 - if you get to your wits' end and want to know how to complain

3 CUSTOMIZING YOUR CALDB SET-UP

The Caldb system is moderately flexible such that some degree of customization of a user's set-up is available. However such customization is only recommended for advanced users who fully understand the implications. It is intended that the standard set-up described in Section 2 will satisfy the needs of 90% of general users.

3.1 When to Customize

Essentially, the customization involves construction a personal copy of one or more CIFs in order to be able to add to or replace standard calibration datasets with personal copies. Following the customization, these CIFs will be used instead of the standard CIFs available as part of the local Caldb. This is achieved by re-defining the environment variable (or logicals) by which the caldb software locates the relevant files such as to point to some/all files not in a user's official local Caldb. Users effectively become local Caldb managers of a personal Caldb (though they may share some - but not all - datasets with their official local Caldb).

Such customization is recommended only for users who

have a detailed knowledge of the calibration datasets, and wish to investigate the effect on their scientific results of using personal calibration (or developmental calibration datasets not yet released to the community)
wish to use standard calibration datasets not available via their local Caldb (if, for instance, their local Caldb does not include files for a given mission/instrument).

All other users are strongly advised to use the standard set-up described in Section 2.

3.2 How to Customize

As stated in Section 2:

the path to the top of the local caldb directory tree is defined by the CALDB environment variable (logical)
all the environment variables (or logicals) pointing to the CIFs etc are defined by sourcing a file defined by the CALDBCONFIG environment variable (logical).

Under the standard set-up, both CALDB and CALDBCONFIG are defined in the caldbinit file (see Section 2.1).

section incomplete - contact a Team-CALDB member for details

4 CALDB ACCESS TASKS & UTILITIES

section incomplete

5 COMMON PROBLEMS & FAQs

5.1 I don't understand anything

No problem, we'll help you. Don't be shy, just contact us via one of the routes given in Section 6.

Please do, since if we don't hear about your problems, we're unlikely to be able to fix them. Most likely they are trivial and the result of the documentation being insufficient or ambiguous.

5.2 Do I have FTOOLS?

All the Caldb access software is distributed through the FTOOLS software package. Again, the reader should verify that indeed FTOOLS are available at their institution and determine its location. The easiest way to check for FTOOLS is to type the following at the operating system prompt:
fhelp ftools

if
NO
the command is not recognized
- then FTOOLS are not available to you. We suggest you ask around locally as to whether anybody knows whether FTOOLS are available anywhere on your system (also check with your system adminstrator).
  - if
    NO
    FTOOLS are not available locally
    - then go to Section 5.3.
  - if
    YES
    you found a local copy of FTOOLS
    - then do whatever is necessary to make them available to your account, and go to Section 1.1.
if
YES
a list & one-line description of all the tools is written to STDOUT,
- then FTOOLS are available to you. Go to Section 1.1.

5.3 How do I get FTOOLS?

The FTOOLS software package is available from the HEASARC via either the WWW at the URL:
http://heasarc.gsfc.nasa.gov/docs/software/ftools/ftools_release.html or from ftp://legacy.gsfc.nasa.gov/software/ftools/release.

Both areas contain the necessary Release Notes and The Installation Instructions.

5.4 Why doesn't the FTOOLS installation set up a local Caldb ?

The FTOOLS installation does NOT automatically include the setting up of a local Caldb. The reasons for this are:

Primarily Disk-space !
The total data holding of the CALDB at NASA/GSFC is many Gbytes. This includes files flagged as both "Good" and "bad" for all the mission/instruments for which the HEASARC has data. It is extremely likely that any given site will only require the "Good" files which constitute only a small fraction of the total data volume. We believe users are best equiped to manage their own disk-space, and anticipate most sites will want to be selective as to what calibration datasets they copy for selected mission/instruments.
The Caldb system allows remote users/sites some flexibility to customize their set-up (see Section 3), allowing then to add personal/locally-produced calibration datasets for both mission/instruments which the HEASARC supports and other mission/instruments.

The inclusion of all possible options within the FTOOLS installation would require far too much interactive i/p during the installation for most users to stomach.

6 IF ALL ELSE FAILS ...

Send e-mail to:
caldbhelp@athena.gsfc.nasa.gov
or contact a member of Team Caldb listed at the URL:http://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_team.html

USEFUL LINKS TO OTHER HTML PAGES

The following useful links are available (in the HTML version of this document only):

File translated from T_EX by T_TH, version 3.67.
On 21 May 2007, 17:40.