OGIP Calibration Memo CAL/GEN/03-001
Automated Delivery of Calibration Data to the CALDB
Lorella Angelini & Mike Corcoran
HEASARC
Last Update: 2003 Sep 23
edited by:
Codes 662
NASA/GSFC,
Greenbelt, MD20771
|
Release | Sections Changed | Brief Notes |
Date | ||
2003 Sep 10 | All | First version |
2003 Sep 23 | All | some clarifications |
2004 Jun | Sec 1 | some more clarifications after the first ingest test |
This document provides a guide for delivering data to the CALDB
staging area for automated release into the public CALDB. This
automated mechanism was first used by the SWIFT mission starting in
2003.
Any project/mission which wishes to use this automated release should
identify a responsible party (the ``Project Calibration Team'') to
provide calibration files into an appropriate directory in the CALDB
staging area. This directory will have to be set up by the CALDB ]
manager who can be identified from the CALDB home page,
http://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_intro.html
The CALDB staging directory structure needs to be a mirror of
the directory structure of the mission's CALDB data directories. The CALDB manager can help set this up for you if necessary.
Each delivery consists in the following :
This document describes what the project should do to
deliver one or more calibration files for one or more particular
instruments, and how to signal to the CALDB manager that the delivery
is available in the staging area for automated release to the public CALDB.
Some general notes:
Unless the data files are ``Primary Calibration Files'' (pcf) which
are to be stored for historical purposes (but which are not meant to
be used by analysis software), the data files are expected to be
stored in subdirectories in the ``Basic Calibration File'' (bcf) or
the ``Calibration Products File'' (cpf) directories (see
``BCF & CPF Calibration File Guidelines''
for CPF/BCF information and guidelines).
For bcf or cpf data,
these data
The above 5 steps are the responsibility of
the Project Calibration Team.
where $CALDB_STAGE is a system environment variable which points to the
location on disk of the CALDB staging area, <mission> is the string which
has been adopted to name the mission, and <instrument> is a
string which has been adopted to describe a given element associated
with this mission. Each mission will have one or more instruments.
An instrument can include common elements such as the entire
spacecraft platform (<instrument>=spacecraft) or an X-ray telescope
(<instrument>=xrt).
Calibration data releases need to be on an
instrument-by-instrument basis.
For each <instrument>
for which new calibration data are being staged, the project
calibration team needs to supply a valid Calibration Index File
(CIF). This CIF needs to include not only the new calibration data
being staged, but also needs to include any other valid
calibration data for that mission and instrument.
It is strongly encouraged that all calibration
data (valid and invalid) which may have been released for that
particular mission and instrument be
included in the CIF. All calibration files included in the CIF must include appropriate CALDB header keywords, including the CAL_QUAL which is used to distinguish valid from invalid calibration data included in the CALDB.
The delivery should include a CALDB index named
caldb.indxYYYYMMDD where YYYYMMDD is the staging date. This file should be placed in a subdirectory of the
$CALDB_STAGE/data/swift/uvot/index/caldb.indx20040617
In the public CALDB this file will be placed in a subdirectory
named index within the appropriate instrument directory. A symbolic link to this file is created as
To keep record within the file of a specific caldb version, all caldb.indx
files provided by the project should include a new keyword CALDBVER. The content of the keyword
should match the string defining the staging date, e.g. CALDBVER='19931220'.
After the calibration data and the index file (one per instrument) have been copied to the CALDB staging area
an email should be sent to
caldbingest@olegacy.gsfc.nasa.gov.
The subject of the e-mail
must contain the string ``caldb update'' followed by a space, followed by the mission name, followed by a space, followed by the staging date in YYYYMMDD format.
For example :
The staging date in the subject must be the same date as given in the CALDBVER and as appended to the end of the caldb.indx filename.
The body of the email should be text defining the calibration files for each instrument included in the delivery.
The instrument block is defined as follows.
For example the email notification for a release of ASCA data for the
SIS and GIS instruments might look like the following:
The caldb index file appropriate for that instrument should reflect
what is changed in caldb :
After the data have been
staged and notification has been received by the CALDB manager, the
CALDB manager will:
Contents
1 Overview: Delivering Data to the CALDB at the HEASARC
2 Calibration Data File Requirements
3 Delivery to the CALDB Staging Area
3.1 Organization of the Staging Area
3.2 Staging
4 Email Notification of the CALDB Manager
5 Changes to the Index file
6 Release to the Public CALDB
1 Overview: Delivering Data to the CALDB at the HEASARC
2 Calibration Data File Requirements
http://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_keywords.html
for a list of required keywords);
3 Delivery to the CALDB Staging Area
3.1 Organization of the Staging Area
The CALDB staging area for a given mission and for a given instrument
is organized following standard CALDB usage as
follows:
$CALDB_STAGE/data/<mission>/<instrument>/
3.2 Staging
After verifying a calibration file
(or calibration files) for release to the CALDB, the Project
Calibration Team data copies calibration data files for each instrument to the appropriate
<instrument> subdirectory in the CALDB staging area.
$CALDB_STAGE/data/<mission>/<instrument>/ directory named index. For example
$CALDB_STAGE/data/<mission>/<instrument>/caldb.indx,
which is the file used by software to access calibration data from the CALDB.
Previous versions of the caldb.indexYYYYMMDD files are kept
within the index subdirectory
so that previous versions of the CALDB can be accessed by software simply by re-assigning the symbolic link to a previous version of the <index>/caldb.indxYYYYMMDD file.
4 Email Notification of the CALDB Manager
To: caldbingest@olegacy.gsfc.nasa.gov
Subject: caldb update asca 19931220
indicates that the ASCA calibration data file were staged on December 20 1993.
There should be no blank lines within an instrument block. A release can include more than one instrument blocks.
From: ASCA calibration team
To: caldbingest@olegacy.gsfc.nasa.gov
Subject: caldb update asca 19931220
instrument= gis
caldbindexfile= caldb.indx931220
newfile
cpf/95mar06/gis2v4_0_256.rmf
bcf/bgd/94may/blanksky_g2_10cor12_v2.evt
endnewfile
instrument= sis
caldbindexfile= caldb.indx931220
newfile
cpf/94nov9/s1c3g0234p40e1_512_1av0_8i.rsp
cpf/94nov9/s0c2g0234p40e0_1024v0_8i.rmf
bcf/bgd/94nov/s1bgd_10i.evt
endnewfile
5 Changes to the Index file
6 Release to the Public CALDB
Subject: COPIED caldb update asca 19931220 files