OGIP Calibration Memo CAL/GEN/03-001

Automated Delivery of Calibration Data to the CALDB

edited by:
Lorella Angelini & Mike Corcoran
Codes 662
Greenbelt, MD20771
Last Update: 2003 Sep 23


This documents how to deliver CALDB data files into the CALDB staging area for automatic release to the public CALDB
Intended audience: General users, Data producers, OGIP programmers and authors of data analysis s/w.


Release Sections Changed Brief Notes
2003 Sep 10All First version
2003 Sep 23All some clarifications
2004 Jun Sec 1 some more clarifications after the first ingest test


1  Overview: Delivering Data to the CALDB at the HEASARC
2  Calibration Data File Requirements
3  Delivery to the CALDB Staging Area
    3.1  Organization of the Staging Area
    3.2  Staging
4  Email Notification of the CALDB Manager
5  Changes to the Index file
6  Release to the Public CALDB

1  Overview: Delivering Data to the CALDB at the HEASARC

This document provides a guide for delivering data to the CALDB staging area for automated release into the public CALDB. This automated mechanism was first used by the SWIFT mission starting in 2003. Any project/mission which wishes to use this automated release should identify a responsible party (the "Project Calibration Team") to provide calibration files into an appropriate directory in the CALDB staging area. This directory will have to be set up by the CALDB ] manager who can be identified from the CALDB home page,
The CALDB staging directory structure needs to be a mirror of the directory structure of the mission's CALDB data directories. The CALDB manager can help set this up for you if necessary.
Each delivery consists in the following :
This document describes what the project should do to deliver one or more calibration files for one or more particular instruments, and how to signal to the CALDB manager that the delivery is available in the staging area for automated release to the public CALDB.
Some general notes:

2  Calibration Data File Requirements

Unless the data files are "Primary Calibration Files" (pcf) which are to be stored for historical purposes (but which are not meant to be used by analysis software), the data files are expected to be stored in subdirectories in the "Basic Calibration File" (bcf) or the "Calibration Products File" (cpf) directories (see "BCF & CPF Calibration File Guidelines" for CPF/BCF information and guidelines).
For bcf or cpf data, these data
  1. must be in FITS format and include all required keywords (see
    http://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_keywords.html for a list of required keywords);
  2. the data files must be verified as valid FITS files (using a routine like the fverify tool which is part of the HEASOFT package)
  3. must contain valid calibration data;
  4. the FITS data must contain valid CHECKSUM and DATASUM keywords in each FITS header;
  5. each file should be associated with only 1 CALDB subdirectory.
The above 5 steps are the responsibility of the Project Calibration Team.

3  Delivery to the CALDB Staging Area

3.1  Organization of the Staging Area

The CALDB staging area for a given mission and for a given instrument is organized following standard CALDB usage as follows:
where $CALDB_STAGE is a system environment variable which points to the location on disk of the CALDB staging area, <mission> is the string which has been adopted to name the mission, and <instrument> is a string which has been adopted to describe a given element associated with this mission. Each mission will have one or more instruments. An instrument can include common elements such as the entire spacecraft platform (<instrument>=spacecraft) or an X-ray telescope (<instrument>=xrt). Calibration data releases need to be on an instrument-by-instrument basis.

3.2  Staging

After verifying a calibration file (or calibration files) for release to the CALDB, the Project Calibration Team data copies calibration data files for each instrument to the appropriate <instrument> subdirectory in the CALDB staging area.
For each <instrument> for which new calibration data are being staged, the project calibration team needs to supply a valid Calibration Index File (CIF). This CIF needs to include not only the new calibration data being staged, but also needs to include any other valid calibration data for that mission and instrument. It is strongly encouraged that all calibration data (valid and invalid) which may have been released for that particular mission and instrument be included in the CIF. All calibration files included in the CIF must include appropriate CALDB header keywords, including the CAL_QUAL which is used to distinguish valid from invalid calibration data included in the CALDB.
The delivery should include a CALDB index named caldb.indxYYYYMMDD where YYYYMMDD is the staging date. This file should be placed in a subdirectory of the
$CALDB_STAGE/data/<mission>/<instrument>/ directory named index. For example
In the public CALDB this file will be placed in a subdirectory named index within the appropriate instrument directory. A symbolic link to this file is created as
which is the file used by software to access calibration data from the CALDB. Previous versions of the caldb.indexYYYYMMDD files are kept within the index subdirectory so that previous versions of the CALDB can be accessed by software simply by re-assigning the symbolic link to a previous version of the <index>/caldb.indxYYYYMMDD file.
To keep record within the file of a specific caldb version, all caldb.indx files provided by the project should include a new keyword CALDBVER. The content of the keyword should match the string defining the staging date, e.g. CALDBVER='19931220'.

4  Email Notification of the CALDB Manager

After the calibration data and the index file (one per instrument) have been copied to the CALDB staging area an email should be sent to caldbingest@olegacy.gsfc.nasa.gov. The subject of the e-mail must contain the string "caldb update" followed by a space, followed by the mission name, followed by a space, followed by the staging date in YYYYMMDD format.
For example :
           To: caldbingest@olegacy.gsfc.nasa.gov
   Subject: caldb update asca 19931220

indicates that the ASCA calibration data file were staged on December 20 1993.
The staging date in the subject must be the same date as given in the CALDBVER and as appended to the end of the caldb.indx filename.
The body of the email should be text defining the calibration files for each instrument included in the delivery. The instrument block is defined as follows.
  1. The instrument block begins with the line instrument= followed by the string designating the instrument, for example instrument=gis. There should be no spaces in this line
  2. the next line starts with caldbindexfile=, which gives the name of the CIF appropriate for the particular data delivery. For example caldbindexfile=caldb.indx19931220
  3. The next line contains the tag newfile.
  4. After the newfile tag comes a list of calibration files, one per line. The calibration file names must include the directory path starting from the appropriate subdirectory (the cpf or bcf) of the <instrument> directory. For example cpf/95mar06/gis2v4_0_256.rmf
  5. The endnewfile tag follows the last calibration file included in this particular delivery for this particular instrument and marks the end of the instrument block.
There should be no blank lines within an instrument block. A release can include more than one instrument blocks.
For example the email notification for a release of ASCA data for the SIS and GIS instruments might look like the following:
From: ASCA calibration team
To: caldbingest@olegacy.gsfc.nasa.gov
Subject:  caldb update asca 19931220

    caldbindexfile= caldb.indx931220

5  Changes to the Index file

The caldb index file appropriate for that instrument should reflect what is changed in caldb :

6  Release to the Public CALDB

After the data have been staged and notification has been received by the CALDB manager, the CALDB manager will:
  1. Copy the data files to the appropriate subdirectory of the HEASARC (public) CALDB
  2. Verify the data files which have been copied using the CHECKSUM/DATASUM keywords
  3. Send a confirmation e-mail to the Project Calibration Team that the data have been copied. The body of this e-mail will consist of a list of the copied files in the same format as used in the notification e-mail. The subject line of this e-mail will include the name of the CALDB update, as in:
    Subject: COPIED caldb update asca 19931220 files
  4. The privileges on the copied files in the HEASARC CALDB will be changed to allow world read access.

File translated from TEX by TTH, version 3.84.
On 24 Sep 2008, 10:16.