\documentstyle[11pt,html]{article}
\setlength{\parindent}{0pt} % No indent at start of paragraphs
\setlength{\parskip}{\baselineskip} % Blank line between paragraphs
\setlength{\textwidth}{16cm} % Text width
\setlength{\textheight}{22cm} % Text height
\setlength{\oddsidemargin}{-5mm} % LH margin width
%\setlength{\evensidemargin}{0pt} % LH margin width
\pagestyle{myheadings}
\markboth{}{OGIP Calibration Memo CAL/GEN/03-001}
\setlength{\topmargin}{.5cm}
\setlength{\headsep}{10mm}
\setcounter{totalnumber}{10} % Max no. floats allowed per page
\setcounter{topnumber}{3} % Max no. floats allowed at top
\setcounter{bottomnumber}{3} % Max no. floats allowed at bottom
\begin{document}
\pagenumbering{roman}
OGIP Calibration Memo CAL/GEN/03-001
\vspace{1cm}
\begin{center}
\Large
\Huge
{\bf Automated Delivery of Calibration Data to the CALDB}
\vspace{2cm}
\Large
edited by:
\normalsize
Lorella Angelini \& Mike Corcoran
\normalsize
HEASARC\\
Codes 662 \\
NASA/GSFC, \\
Greenbelt, MD20771
\normalsize
Last Update: 2003 Sep 23 \\
\end{center}
\vspace{2cm}
\section*{SUMMARY}
\begin{minipage}[t]{13cm}
This documents how to deliver CALDB data files
into the CALDB staging area for automatic release to the public CALDB
Intended audience: General users, Data producers,
OGIP programmers and authors of data analysis s/w.
\end{minipage}
\newpage
\section*{LOG OF SIGNIFICANT CHANGES}
\typeout{Changes Log}
\begin{table}[h]
\centering
\vspace{5mm}
\label{tab:changes_log}
\begin{tabular}{|r|l|l|} \hline
\multicolumn{1}{|c|}{\bf Release} &
\multicolumn{1}{|c|}{\bf Sections Changed} &
\multicolumn{1}{|c|}{\bf Brief Notes}\\
\multicolumn{1}{|c|}{\bf Date} & & \\ \hline
& & \\
2003 Sep 10& All & First version\\
2003 Sep 23& All & some clarifications\\
2004 Jun & Sec 1 & some more clarifications after the first ingest test\\ \hline
\end{tabular}
\end{table}
\normalsize
\newpage
\tableofcontents
\newpage
\pagenumbering{arabic}
%\renewcommand{\thepage}{\arabic{chapter}-\arabic{page}}
\setcounter{page}{1}
%%\chapter{INTRODUCTION}
%\label{Chpt:intro}
\section{Overview: Delivering Data to the \href{/docs/heasarc/caldb}{CALDB} at the \href{http://heasarc.gsfc.nasa.gov}{HEASARC}}
This document provides a guide for delivering data to the CALDB
staging area for automated release into the public CALDB. This
automated mechanism was first used by the SWIFT mission starting in
2003.
%CALDB now support an automatic way to make the CALDB files public.
Any project/mission which wishes to use this automated release should
identify a responsible party (the ``Project Calibration Team'') to
provide calibration files into an appropriate directory in the CALDB
staging area. This directory will have to be set up by the CALDB ]
manager who can be identified from the CALDB home page,\\
\url{http://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_intro.html}
%\begin{htmlonly}
%\begin{rawhtml}
%(click
%here)
%\end{rawhtml}
%\end{htmlonly}
\noindent
The CALDB staging directory structure needs to be a mirror of
the directory structure of the mission's CALDB data directories. The CALDB manager can help set this up for you if necessary.
Each delivery consists in the following :
\begin{itemize}
\item One or more Calibration data files copied to the CALDB staging
area in the appropriate subdirectory
\item A Calibration Index File
\item An email sent to the \verb+caldbingest@olegacy.gsfc.nasa.gov+ address.
\end{itemize}
This document describes what the project should do to
deliver one or more calibration files for one or more particular
instruments, and how to signal to the CALDB manager that the delivery
is available in the staging area for automated release to the public CALDB.
Some general notes:
\begin{itemize}
\item To ensure consistency with different flavors of unix and other supported operating systems, the CALDB requires that missions, instruments and file names are in \textbf{lower case}.
\item Mission and instrument names should not contain special characters and should be consistent with the specifications of \href{/docs/heasarc/ofwg/docs/general/ogip_93_013/ogip_93_013.html}{OGIP/93-013: "Standard Strings for Mission, Instrument, Filter, Detector \& Grating Names for OGIP FITS files"}.
\item File names can consist of letters, numbers and underscores.
\item to write to the CALDB staging area at the HEASARC the user must be added to the \texttt{caldb\_stage} group. Contact the \href{mailto:caldbingest@olegacy.gsfc.nasa.gov}{CALDB manager} for instructions.
\end{itemize}
\section{Calibration Data File Requirements}
Unless the data files are ``Primary Calibration Files'' (pcf) which
are to be stored for historical purposes (but which are not meant to
be used by analysis software), the data files are expected to be
stored in subdirectories in the ``Basic Calibration File'' (bcf) or
the ``Calibration Products File'' (cpf) directories (see
\href{/docs/heasarc/caldb/docs/summary/cal_gen_92_003_summary.html}{``BCF \& CPF Calibration File Guidelines''}
for CPF/BCF information and guidelines).
For bcf or cpf data,
these data
\begin{enumerate}
\item must be in FITS format and include all required keywords (see
\\
\verb+http://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/caldb_keywords.html+
for a list of required keywords);
\item the data files must be
verified as valid FITS files (using a routine like the \verb+fverify+
tool which is part of the HEASOFT package)
\item must contain valid
calibration data;
\item the FITS data must contain valid CHECKSUM and
DATASUM keywords in each FITS header;
\item each file should be
associated with only 1 CALDB subdirectory.
\end{enumerate}
\textbf{The above 5 steps are the responsibility of
the Project Calibration Team.}
\section{Delivery to the CALDB Staging Area}
\subsection{Organization of the Staging Area}
The CALDB staging area for a given mission and for a given instrument
is organized following standard CALDB usage as
follows:\\
\verb+$CALDB_STAGE/data///+
where \verb+$CALDB_STAGE+ is a system environment variable which points to the
location on disk of the CALDB staging area, \verb++ is the string which
has been adopted to name the mission, and \verb++ is a
string which has been adopted to describe a given element associated
with this mission. Each mission will have one or more instruments.
An instrument can include common elements such as the entire
spacecraft platform (\verb+=spacecraft+) or an X-ray telescope
(\verb+=xrt+).
%which may be shared by two or more
%sub-components.
Calibration data releases need to be on an
instrument-by-instrument basis.
\subsection{Staging}
After verifying a calibration file
(or calibration files) for release to the CALDB, the Project
Calibration Team data copies calibration data files for each instrument to the appropriate
\verb++ subdirectory in the CALDB staging area.
For each \verb++
for which new calibration data are being staged, the project
calibration team needs to supply a valid \textbf{Calibration Index File}
(CIF). This CIF needs to include not only the new calibration data
being staged, but also needs to include any other valid
calibration data for that mission and instrument.
It is strongly encouraged that all calibration
data (valid and invalid) which may have been released for that
particular mission and instrument be
included in the CIF. All calibration files included in the CIF must include appropriate \href{/docs/heasarc/caldb/caldb_keywords.html}{CALDB header keywords}, including the \verb+CAL_QUAL+ which is used to distinguish valid from invalid calibration data included in the CALDB.
% For each data delivery from a given mission for a given instrument,
%an index file
% should be present for each of the
% instruement. The index file should be updated with all files included
% in caldb, even if the delivery includes only updates or new files
% (see below Delivery section). Within the index file the QUALITY column
% should be properly flagged for all data files in CALDB.
The delivery should include a CALDB index named
%as follows:
%\begin{itemize}
%\item caldb.indx. This file is used by the CALDB software to query
%the CALDB and access/retrieve data from the CALDB.
%\item
\verb+caldb.indxYYYYMMDD+ where \verb+YYYYMMDD+ is the staging date. This file should be placed in a subdirectory of the \\
\verb+$CALDB_STAGE/data///+ directory named \texttt{index}. For example
\noindent
\verb+$CALDB_STAGE/data/swift/uvot/index/caldb.indx20040617+
In the public CALDB this file will be placed in a subdirectory
named \verb+index+ within the appropriate \verb+instrument+ directory. A symbolic link to this file is created as
\\
\verb+$CALDB_STAGE/data///caldb.indx+, \\
which is the file used by software to access calibration data from the CALDB.
Previous versions of the \verb+caldb.indexYYYYMMDD+ files are kept
within the \verb+index+ subdirectory
%to allow to recovery of information
%on what files were present in CALDB and what was their quality
%setting in each release.
%In this way files produced with a specific delivery of
so that previous versions of the CALDB can be accessed by software simply by re-assigning the symbolic link to a previous version of the \verb+/caldb.indxYYYYMMDD+ file.
% reproduced by selecting the specific \texttt{caldb.indxYYYYMMDD}.
%\end{itemize}
To keep record within the file of a specific caldb version, all \texttt{caldb.indx}
files provided by the project should include a new keyword \texttt{CALDBVER}. The content of the keyword
should match the string defining the staging date, e.g. \verb+CALDBVER='19931220'+.
\section{Email Notification of the CALDB Manager}
After the calibration data and the index file (one per instrument) have been copied to the CALDB staging area
an email should be sent to
\verb+caldbingest@olegacy.gsfc.nasa.gov+.
The subject of the e-mail
must contain the string ``caldb update'' followed by a space, followed by the mission name, followed by a space, followed by the staging date in YYYYMMDD format.
For example :
\begin{verbatim}
To: caldbingest@olegacy.gsfc.nasa.gov
Subject: caldb update asca 19931220
\end{verbatim}
indicates that the ASCA calibration data file were staged on December 20 1993.
The staging date in the subject must be the same date as given in the \texttt{CALDBVER} and as appended to the end of the \texttt{caldb.indx} filename.
The body of the email should be text defining the calibration files for each instrument included in the delivery.
The \verb+instrument block+ is defined as follows.
\begin{enumerate}
\item The \verb+instrument block+ begins with the line \verb+instrument=+ followed by the string designating the instrument, for example \verb+instrument=gis+. There should be no spaces in this line
\item the next line starts with \verb+caldbindexfile=+, which gives the name
of the CIF appropriate for the particular data delivery. For example \verb+caldbindexfile=caldb.indx19931220+
\item The next line contains the tag \verb+newfile+.
\item After the \verb+newfile+ tag comes a list of calibration files, one per line. The calibration file names must include the directory path starting from the appropriate subdirectory (the \verb+cpf+ or
\verb+bcf+) of the \verb++ directory. For example \verb+cpf/95mar06/gis2v4_0_256.rmf+
\item The \verb+endnewfile+ tag follows the last calibration file included in this particular delivery for this particular instrument and marks the end of the instrument block.
\end{enumerate}
There should be no blank lines within an \verb+instrument block+. A release can include more than one \verb+instrument blocks+.
%and
%\verb+caldbindexfile+) and the list of files included
% between a set of markers.
%The keywords are 'instrument' and 'caldbindexfile' and
% should be set to the instrument name and the caldbindex file for that
% instrument respectively.
% The strings for the markers are 'newfile' and 'endnewfile'. After
% 'newfile' starts the list of files given one per line with the directory
% structure. The list should end with the string 'endnewfile'.
% If the delivery includes data file from different instrument of the same
% mission the above structure should be repeted within the same e-mail
% for each of the instrument.
For example the email notification for a release of ASCA data for the
SIS and GIS instruments might look like the following:
\begin{verbatim}
From: ASCA calibration team
To: caldbingest@olegacy.gsfc.nasa.gov
Subject: caldb update asca 19931220
instrument=gis
caldbindexfile=caldb.indx931220
newfile
cpf/95mar06/gis2v4_0_256.rmf
bcf/bgd/94may/blanksky_g2_10cor12_v2.evt
endnewfile
instrument=sis
caldbindexfile= caldb.indx931220
newfile
cpf/94nov9/s1c3g0234p40e1_512_1av0_8i.rsp
cpf/94nov9/s0c2g0234p40e0_1024v0_8i.rmf
bcf/bgd/94nov/s1bgd_10i.evt
endnewfile
\end{verbatim}
\section{Changes to the Index file}
%The first delivery should contain all the CALDB files.
%Updates
% either because a new version of the file has been released
% or because a new file has been added should contain ONLY the new data
% file.
The caldb index file appropriate for that instrument should reflect
what is changed in caldb :
\begin{itemize}
\item Delivery of a new calibration file: The caldb index file should include entries which
document the new files.
\item Delivery of an updated file: The caldb index file should include new entries
documenting the updated file. The \verb+CAL_QUAL+ value for the old
file should be set to a higher value (usually changed from
\verb+CAL_QUAL=0+ to \verb+CAL_QUAL=5+, while the update should have \verb+CAL_QUAL=0+.
\end{itemize}
\section{Release to the Public CALDB}
After the data have been
staged and notification has been received by the CALDB manager, the
CALDB manager will:
\begin{enumerate}
\item Copy the data files to the appropriate subdirectory of the
HEASARC (public) CALDB
\item Verify the data files which have been
copied using the \verb+CHECKSUM/DATASUM+ keywords
\item Send a
confirmation e-mail to the Project Calibration Team that the data
have been copied. The body of this e-mail will consist of a list of
the copied files in the same format as used in the notification
e-mail. The subject line of this e-mail will include the name of the
CALDB update, as in:
\begin{verbatim}
Subject: COPIED caldb update asca 19931220 files
\end{verbatim}
\item The privileges on the copied files in the HEASARC CALDB will be
changed to allow world read access.
\end{enumerate}
\end{document}