Contents
1 Overview: Delivering Data to the CALDB at the HEASARC
2 Calibration Data File Requirements
3 Delivery to the CALDB Staging Area
3.1 Organization of the Staging Area
3.2 Staging
4 Email Notification of the CALDB Manager
5 Changes to the Index file
6 Release to the Public CALDB
1 Overview: Delivering Data to the CALDB at the HEASARC
This document provides a guide for delivering data to the CALDB
staging area for automated release into the public CALDB. This
automated mechanism was first used by the SWIFT mission starting in
2003.
Any project/mission which wishes to use this automated release should
identify a responsible party (the "Project Calibration Team") to
provide calibration files into an appropriate directory in the CALDB
staging area. This directory will have to be set up by the CALDB ]
manager who can be identified from the CALDB home page,
http://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_intro.html
The CALDB staging directory structure needs to be a mirror of
the directory structure of the mission's CALDB data directories. The CALDB manager can help set this up for you if necessary.
Each delivery consists in the following :
- One or more Calibration data files copied to the CALDB staging
area in the appropriate subdirectory
- A Calibration Index File
- An email sent to the caldbingest@olegacy.gsfc.nasa.gov address.
This document describes what the project should do to
deliver one or more calibration files for one or more particular
instruments, and how to signal to the CALDB manager that the delivery
is available in the staging area for automated release to the public CALDB.
Some general notes:
2 Calibration Data File Requirements
Unless the data files are "Primary Calibration Files" (pcf) which
are to be stored for historical purposes (but which are not meant to
be used by analysis software), the data files are expected to be
stored in subdirectories in the "Basic Calibration File" (bcf) or
the "Calibration Products File" (cpf) directories (see
"BCF & CPF Calibration File Guidelines"
for CPF/BCF information and guidelines).
For bcf or cpf data,
these data
- must be in FITS format and include all required keywords (see
http://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_keywords.html
for a list of required keywords);
- the data files must be
verified as valid FITS files (using a routine like the fverify
tool which is part of the HEASOFT package)
- must contain valid
calibration data;
- the FITS data must contain valid CHECKSUM and
DATASUM keywords in each FITS header;
- each file should be
associated with only 1 CALDB subdirectory.
The above 5 steps are the responsibility of
the Project Calibration Team.
3 Delivery to the CALDB Staging Area
3.1 Organization of the Staging Area
The CALDB staging area for a given mission and for a given instrument
is organized following standard CALDB usage as
follows:
$CALDB_STAGE/data/<mission>/<instrument>/
where $CALDB_STAGE is a system environment variable which points to the
location on disk of the CALDB staging area, <mission> is the string which
has been adopted to name the mission, and <instrument> is a
string which has been adopted to describe a given element associated
with this mission. Each mission will have one or more instruments.
An instrument can include common elements such as the entire
spacecraft platform (<instrument>=spacecraft) or an X-ray telescope
(<instrument>=xrt).
Calibration data releases need to be on an
instrument-by-instrument basis.
3.2 Staging
After verifying a calibration file
(or calibration files) for release to the CALDB, the Project
Calibration Team data copies calibration data files for each instrument to the appropriate
<instrument> subdirectory in the CALDB staging area.
For each <instrument>
for which new calibration data are being staged, the project
calibration team needs to supply a valid Calibration Index File
(CIF). This CIF needs to include not only the new calibration data
being staged, but also needs to include any other valid
calibration data for that mission and instrument.
It is strongly encouraged that all calibration
data (valid and invalid) which may have been released for that
particular mission and instrument be
included in the CIF. All calibration files included in the CIF must include appropriate CALDB header keywords, including the CAL_QUAL which is used to distinguish valid from invalid calibration data included in the CALDB.
The delivery should include a CALDB index named
caldb.indxYYYYMMDD where YYYYMMDD is the staging date. This file should be placed in a subdirectory of the
$CALDB_STAGE/data/<mission>/<instrument>/ directory named index. For example
$CALDB_STAGE/data/swift/uvot/index/caldb.indx20040617
In the public CALDB this file will be placed in a subdirectory
named index within the appropriate instrument directory. A symbolic link to this file is created as
$CALDB_STAGE/data/<mission>/<instrument>/caldb.indx,
which is the file used by software to access calibration data from the CALDB.
Previous versions of the caldb.indexYYYYMMDD files are kept
within the index subdirectory
so that previous versions of the CALDB can be accessed by software simply by re-assigning the symbolic link to a previous version of the <index>/caldb.indxYYYYMMDD file.
To keep record within the file of a specific caldb version, all caldb.indx
files provided by the project should include a new keyword CALDBVER. The content of the keyword
should match the string defining the staging date, e.g. CALDBVER='19931220'.
4 Email Notification of the CALDB Manager
After the calibration data and the index file (one per instrument) have been copied to the CALDB staging area
an email should be sent to
caldbingest@olegacy.gsfc.nasa.gov.
The subject of the e-mail
must contain the string "caldb update" followed by a space, followed by the mission name, followed by a space, followed by the staging date in YYYYMMDD format.
For example :
To: caldbingest@olegacy.gsfc.nasa.gov
Subject: caldb update asca 19931220
indicates that the ASCA calibration data file were staged on December 20 1993.
The staging date in the subject must be the same date as given in the CALDBVER and as appended to the end of the caldb.indx filename.
The body of the email should be text defining the calibration files for each instrument included in the delivery.
The instrument block is defined as follows.
- The instrument block begins with the line instrument= followed by the string designating the instrument, for example instrument=gis. There should be no spaces in this line
- the next line starts with caldbindexfile=, which gives the name
of the CIF appropriate for the particular data delivery. For example caldbindexfile=caldb.indx19931220
- The next line contains the tag newfile.
- After the newfile tag comes a list of calibration files, one per line. The calibration file names must include the directory path starting from the appropriate subdirectory (the cpf or
bcf) of the <instrument> directory. For example cpf/95mar06/gis2v4_0_256.rmf
- The endnewfile tag follows the last calibration file included in this particular delivery for this particular instrument and marks the end of the instrument block.
There should be no blank lines within an instrument block. A release can include more than one instrument blocks.
For example the email notification for a release of ASCA data for the
SIS and GIS instruments might look like the following:
From: ASCA calibration team
To: caldbingest@olegacy.gsfc.nasa.gov
Subject: caldb update asca 19931220
instrument=gis
caldbindexfile=caldb.indx931220
newfile
cpf/95mar06/gis2v4_0_256.rmf
bcf/bgd/94may/blanksky_g2_10cor12_v2.evt
endnewfile
instrument=sis
caldbindexfile= caldb.indx931220
newfile
cpf/94nov9/s1c3g0234p40e1_512_1av0_8i.rsp
cpf/94nov9/s0c2g0234p40e0_1024v0_8i.rmf
bcf/bgd/94nov/s1bgd_10i.evt
endnewfile
5 Changes to the Index file
The caldb index file appropriate for that instrument should reflect
what is changed in caldb :
- Delivery of a new calibration file: The caldb index file should include entries which
document the new files.
- Delivery of an updated file: The caldb index file should include new entries
documenting the updated file. The CAL_QUAL value for the old
file should be set to a higher value (usually changed from
CAL_QUAL=0 to CAL_QUAL=5, while the update should have CAL_QUAL=0.
6 Release to the Public CALDB
After the data have been
staged and notification has been received by the CALDB manager, the
CALDB manager will:
- Copy the data files to the appropriate subdirectory of the
HEASARC (public) CALDB
- Verify the data files which have been
copied using the CHECKSUM/DATASUM keywords
- Send a
confirmation e-mail to the Project Calibration Team that the data
have been copied. The body of this e-mail will consist of a list of
the copied files in the same format as used in the notification
e-mail. The subject line of this e-mail will include the name of the
CALDB update, as in:
Subject: COPIED caldb update asca 19931220 files
- The privileges on the copied files in the HEASARC CALDB will be
changed to allow world read access.