OGIP Calibration Memo CAL/SW/95-002

OGIP CALDB Access Subroutines
(a brief overview for programmers)
Ian M. George,
Lorraine Breedon,
Michael F. Corcoran

Mail Code 662
Greenbelt, MD20771
Version: 2006 Aug 21
(Previous Versions: 1996 Jan 02)


This document provides an introduction of the FORTRAN subroutines, supplied as part of the FTOOLS package, which can be called by tasks in order to access their local copy of the HEASARC's Calibration Database (Caldb). Also included is a brief review of the HEASARC Caldb, the preliminaries a user needs to have completed prior to using any task which calls the Caldb-access subroutines, along with programming suggestions/examples.
Intended Audience: Programmers wishing to make use of the their local Caldb.


This document is intended to provide a general introduction to HEASARC's calibration database system access software. Every attempt has been made to present the information in a clear, and (moderately) concise manner, and copious use of cross-referencing is made, although reference to other Calibration Memos etc is occasionally required.
At the current time, this document is updated as necessary, and not guaranteed complete.
All questions, comments and suggestions should be directed to:
or Caldb personnel as listed on the Caldb Personnel Page.
The latest version of this document in Portable Document Format can be obtained at


Release Sections Changed Brief Notes
1994 Sep 12 Original Version
1995 Jan 11 All Made compatible with LaTeX2HTML software
1995 Oct 18 All Updated & made specific to subroutines only
1996 Jan 02 4.1 Updated to include caldb_info subroutine
2006 Aug 21 all minor typo corrections & updates, including change from gtcal.f to gtcalf.f (MFC)


    3.1  Is the Caldb-access software absolutely necessary ?
    3.2  Allow user input options
    3.3  Report failures back to the user in some detail
    3.4  Include the details of any calibration used in the output file
    4.1  caldb_info.f
    4.2  gtcalf.f
    5.1  What's the best introduction the Caldb ?
        5.1.1  For the Programmer
        5.1.2  For General Users
        5.1.3  For Local Caldb Managers
    5.2  What's the difference between the "HEASARC Caldb" & a "local Caldb"?
    5.3  What should I put in my "help file"?
    5.4  Can I use the ftools/refdata area?
    5.5  What if a local Caldb does not appear to be set-up?
    5.6  What do I do once I've found the Calibration filename?
    5.7  What if a calibration file is not found?
    5.8  What if many calibration files are found (unexpectedly)?
    5.9  Where do I deliver new calibration files to ?
    6.1  Using caldb_info & gtcalf


One of the main goals of the HEASARC's Calibration Database (Caldb) system is to provide analysis software access to relevant calibration datasets, without having to have the names and locations of those datasets hard-wired into the analysis software. "Access" means the capability to identify a calibration data file based on specified instrumental properties or parameters. These parameters or properties may include (but are not limited to) mission and instrument name, detectors and filters in use, observation date and time, and other parameters like temperatures, high voltages, magnetic rigidities, orbital parameters, time since passage through the South Atlantic Anomaly, etc. This allows updates to the calibration datasets to be made immediately accessible without the need for changes to be made to the analysis software.
To achieve these goals, the HEASARC has designed and developed the Calibration Database. This is essentially a simple relational database whereby critical information concerning each and every calibration dataset within the database for a given instrument is stored in a Calibration Index File (CIF), which is a FITS-formatted binary table. The format of the CIFs are described in detail in OGIP memo CAL/GEN/92-008. The Caldb access software described in this document basically interogates the relevant CIF, searching for the entry (ie dataset) required by the application task which matches the observational parameters.
A detailed knowledge of the CIF format is not required here. In essence a CIF consists of a FITS BINTABLE in which each row refers to a different calibration dataset, with necessary information stored in each of the many BINTABLE columns. For the purposes of this brief introduction, the CIF table columns are those containing the name and location of each calibration dataset, the instrument to which it refers, a code describing what kind of calibration information is actually stored within the dataset, and a time/date stamp indicating when the dataset should first be used. These will be explained in more detail below. The information contained within the CIF allows the Caldb access software to determine which dataset(s) are required for a given analysis task, and to return the name & location of the dataset(s) to the task. It is then the responsibility of the application to open, read and use the dataset in an appropriate manner.
This document describes the various Caldb access subroutines available from the HEASARC CALTOOLS software suite, and gives examples on their use.


There are two basic requirements for a user/programmer to be able to use their local Caldb (and the access subroutines described in Section 4):


There are a number of issues which should be born in mind when decided whether/how to include access to a local Caldb within a standalone task.

3.1  Is the Caldb-access software absolutely necessary ?

Obviously the Caldb-access software assumes that the site where the users of your task are running it have (or could have) an up-to-date Caldb set-up, and have access to it. This means that there are some overheads associated with setting up & maintaining a local Caldb. Thus in cases where your task will always require the same highly task-specific file (which will not change with time), then Caldb-access may not be 100% necessary.
However, experience with many missions has highlighted that nearly all calibration files DO change at some point, and thus use of the Caldb structure and access software often saves significant time in the long run.

3.2  Allow user input options

Making an analysis task totally reliant on having a local Caldb set-up and available to users may be as bad as having filenames & locations hard-wired into the code. It is nearly always trivial to structure analysis software so that any local Caldb can be used, or so that the user can manually provide the location & name of the calibration file to be used. This approach allows users to construct their own calibration datasets to be used in place of "standard" files, which is especially important during analysis tool development and testing.
FTOOLS tasks usually allow a user to specify a character-string parameter for each calibration file which is to be input. This input may be a "special string" (usually CALDB) which signifies that files given in the Caldb are to be used; alternatively a user can simply provide the full path to the desired calibration file as the input, bypassing the Caldb.

3.3  Report failures back to the user in some detail

One of the biggest complaints the Caldb gets is that when the Caldb-access software fails, the task from which it is called fails to deliver coherent and/or useful error messages.
The error-reporting of the Caldb-access software is being improved, but the messages will always be less coherent/detailed compared to those which could be constructed by the main task. This is particularly the case when any of the Caldb-access routines are run in ßilent-mode" (see below). Assuming the error is considered fatal by your task, then at the very least, the task should report which type of calibration file the software was attempting to find, and (perhaps) suggest a course action to the user.

3.4  Include the details of any calibration used in the output file

All tasks which obtain & use a calibration dataset via the Caldb-access software should also include the name of the file (and the extension within that file if necessary) somewhere/somehow in the output file. This will provide crucial history information to users should the calibration change at some time in the future. Clearly including this information is also a good programming practice even when the Caldb-access software is not used. In the case of FITS output files, the most obvious way of doing this is via COMMENT or HISTORY cards.


4.1  caldb_info.f

The FORTRAN subroutine caldb_info.f is intended to enable tasks to check and (if desired) report back on whether a local Calibration Database (Caldb) appears to be correctly set-up and accessible to a user. The subroutine is distributed with the FTOOLS software package, and source code can be found in the ftools library as callib/src/gen/caldb_info.f. The caldb_info subroutine can be linked through the libcaltools.a library. Programmers may therefore wish to call caldb_info.f prior to calling other Caldb access routines within a given task. The level of checking and detail of the reporting depend upon the passed parameters mode and chatter as described below.
If a problem with the user's Caldb set-up/access is detected, then an error message will be written to STDERR, and the passed parameter ierr will be returned with a non-zero value.
The calling sequence is as follows:
	call caldb_info(chatter,mode,mission,instrument,ierr)

        character*(*) mission,instrument
        character*(*) mode
        integer chatter, ierr

where the arguments are as follows:

4.2  gtcalf.f

The current subroutine to use for retrieving information regarding CALDB datasets is the FORTRAN subroutine gtcalf.f. This subroutine was previously called gtcal.f. This subroutine is distributed with the FTOOLS] software package, and source code can be found in the ftools source distribution in the directory ftools/callib/src/gen. The gtcalf subroutine can be linked through the libftools.a library.
The gtcalf subroutine returns the location of calibration datasets located in the Calibration Database by referring to the appropriate Calibration Index File (CIF; see CAL/GEN/92-008 for further details). Selection of the appropriate calibration data is based on the values of the arguments mission, instrument, sub-instr, filter, codenam, date, time and expr. These arguments respectively describe the mission or telescope, instrument, sub-detector (if any), filter (if any), type of dataset, date, time, and calibration boundaries for which the returned datasets should be valid (see also below). In addition to the arguments explicitly listed here, this routine also uses the values of the environment variables (logicals) CALDB, and CALDBCONFIG (as defined in Section 2). See the Caldb Users' Guide (CAL/GEN/94-002) for info on how these system variables should be set.
The maximum number of datasets matching the criteria returned is speficied by the maxret argument. Any datasets which meet the selection criteria are returned through the filenam and extno arrays. Each element of the filenam array contains the complete system dependent path (including the filename) to the file where the calibration data resides. The corresponding element of the extno array contains the FITS extension number of the calibration data within the file.
The calling sequence is as follows:
        call gtcalf(chatter,tele_str,instr_str,detnam_str,
     &  filt_str, codenam_str, strtdate, strtime, stpdate, stptime,
     &  expr_str,maxret,filenam,extno,online,
     &  nret,nfound, status)

        character*(*) tele_str, instr_str, detnam_str, filt_str
        character*(*) codenam_str, expr_str
        character*(*) strtdate, strtime, stpdate, stptime
        integer       maxret
        character*(*) filenam(maxret), online(maxret)
        integer       extno(maxret), nret, nfound, status
        integer       chatter

where the arguments are as follows:


5.1  What's the best introduction the Caldb ?

5.1.1  For the Programmer

The Caldb & its access software was designed with the goal of application programmers not having to know too much about the inner-workings of the database itself. Indeed, assuming the calibration file(s) the programmer's application needs to locate is already available in a local Caldb, then this document is intended to contain everything the programmer needs to know in order to find the file (though certainly not how to read & use the file - see SectionSec:now-what).
Besides the general documents listed in Section 5.1.2, programmers wishing to know about the more technical aspects of the Caldb can check-out:

5.1.2  For General Users

5.1.3  For Local Caldb Managers

5.2  What's the difference between the "HEASARC Caldb" & a "local Caldb"?

There are essentially two main components to keep in mind: 1) the Caldb database system (ie the software) written and distributed by the HEASARC, and 2) the calibration data files themselves (ie the database) on-line at the HEASARC. Collectively, these are often referred to as the "HEASARC Caldb". When some/all the calibration data files have been copied to a remote site, and the Caldb database software installed (CIFs made etc), then this is often referred to as a "local Caldb".

5.3  What should I put in my "help file"?

The help file should include at minimum:

5.4  Can I use the ftools/refdata area?

The "refdata" area is a directory of calibration files distributed with the FTOOLS] software package. Use of the refdata area is deprecated, since maintenance and versioning of the data files in this directory is difficult. However, strenuous

5.5  What if a local Caldb does not appear to be set-up?

If a local Caldb does not appear to be set-up at your site, but you anticipate writing tasks which will/can make use of a local Caldb, then we suggest you investigate setting one up. The process is described in CAL/GEN/94-004 and really isn't that difficult. It should also be noted that as a "local Caldb manager", one can choose which mission/instrument combinations you wish to include in your local Caldb, and even which subset of data files. A local Caldb does not have to contain all the calibration files available at the HEASARC. Indeed your local Caldb could even only contain the files ncessary for your task(s).

5.6  What do I do once I've found the Calibration filename?

Use of calibration data is highly application-specific. However, it should be noted that Caldb files conform to strict FITS format guidelines. In most of these cases, there are standard FORTRAN readers & writers distributed via the FTOOLS package (usually in the libcaltools.a library). Thus, it's recommended that you look for such a standard reader and determine whether it is exactly what you were after, or can be used as a template.

5.7  What if a calibration file is not found?

Assuming the calibration file is vital to your task, then clearly the task should gracefully come to a halt (with plenty of explanation to the user explaining what type of file wasn't found - see Section 3.3). There basically three explanations: In all cases, it should be obvious that a coherent explanation to the user will save everybody time.
It should be remembered that in some cases, a failure to find a certain type of calibration file might not be a fatal error. Intelligent tasks might even anticipate such a situation, and do something appropriate (like construct the required calibration dataset for itself). There is nothing wrong in writing tasks like this. However, in such cases, it is probably desirable to run the access software in ßilent mode" so that users aren't confused by apparent error messages.

5.8  What if many calibration files are found (unexpectedly)?

It is certainly valid to write tasks whereby multiple calibration datasets are returned, leaving the task to decide if/how each dataset is to be used. This is achieved by deliberately passing ßloppy" inputs to the access software. Clearly, if the access software unexpectedly claims more than one calibration file satisfies the input criteria, then in most cases the task should gracefully come to a halt. Again, a coherent explanation as to why it is stopping should be supplied to the user.

5.9  Where do I deliver new calibration files to ?

Send e-mail to:
or to Caldb personne listed on the Caldb Personnel Page. with as many details as possible regarding the dataset (what it's for; where it came from; etc.) during development of the task. They will then discuss the format with you, perhaps suggest additional keywords and/or updates to existing keyword values. Finally they will instruct you how to formally deliver the file.


6.1  Using caldb_info & gtcalf

Example 1

In the following hunk of FORTRAN, it is assumed that the user has entered a character string (calfexp) for the calibration file to be used.
C Check to see whether CALDB access software is to be used
      if((calfexp(1:5).eq.'CALDB').or.(calfexp.eq.'caldb')) then
c ....... Yes, the local CALDB is to be used, so check that it is available
          infomode = 'INST'              ! check CALDB OK this mission/instrument
          infochatter = 0                ! Run in silent mode
          call caldb_info(infochatter,infomode,mission,instrument,ierr)
          if(ierr.NE.0) then
            message = 'CALDB not defined/available'
            call wterrm(subname, version,message)
            message = 'Task requires CALDB to be both defined '//
     &          '& available in order to run'
            call wtinfo(chatter,1,1, message)
            goto 999                      ! Error out     
c ....... Now call the caldb access software 
          chatter = 0                      ! Run in silent mode
          tele_str= 'XTE'
          instr_str = 'PCA'                ! Mission Name
          detnam = '-'                     ! DETECTOR name
          filter = '-'                     ! Not applicable for this instrument
          codenam_str = 'COLLRESP'         ! Looking for a collimator response
          expr_str = 'energy.eq.10'        ! Want dataset valid at 10 keV
          maxret = 1                       ! Only expect 1 dataset to be returned
          call gtcalf(chatter,tele_str,instr_str,detnam_str,
       &  filt_str, codenam_str, strtdate, strtime, stpdate, stptime,
       &  expr_str,maxret,calfilename,extno,online,
       &  nret,nfound, status)
          if(status.NE.0) then
            message = 'Problem obtaining valid COLLRESP dataset '//
     & 	        'valid at 10 keV from the CALDB'
            call wterrm(subname, version,message)
            goto 999                      ! Error out     
          elseif(nfound.gt.1) then
            message = 'More than one valid COLLRESP dataset '//
     & 	        'valid at 10 keV found in the CALDB'
            call wterrm(subname, version,message)
            message = 'Unable to determine which to use -- aborting'
            call wtinfo(chatter,1,1, message)
            ierr = 1
            goto 999                      ! Error out     
c ....... local CALDB is not to be used, rather the user has apparently entered
c         the path/name of the calibration file themselves - so translate its
c         name (taking off any extension specified by the user).
         call fcpars(calfexp, calfilename(1),calextno(1),ierr)
          if(ierr.NE.0) then
            message = 'Problem parsing calfexp experession'
            call wterrm(subname, version,message)
            goto 999                      ! Error out     
c ... So, now
c       calfilename(1) contains the name of the calibration file to be opened
c       calextno(1) contains ext# of dataset (or -99 indicating search reqd)
c Open i/p file
        status = 0
        call cgetlun(iunit)
        call ftopen(iunit,calfilename(1),0,block,ierr)
        if(ierr.ne.0) then
           message = 'Problem opening '//calfilename(1)(:clenact(calfilename(1))
           call wtferr(subname, version, ierr, message)
           goto 999                      ! Error out     

c                           ..... etc etc .....

File translated from TEX by TTH, version 3.73.
On 21 Aug 2006, 12:24.