OGIP Calibration Memo CAL/GEN/92-015

How to Manage a Calibration Database

Lorraine Breedon

Code 668,
NASA/GSFC,
Greenbelt, MD 20771

Version: 1997 Jun 20






SUMMARY

This document describes the steps required to manage a local Calibration Database (Caldb) system . The local Caldb can be an exact copy of the Caldb at the HEASARC or a `customised' caldb containing HEASARC supported missions but with personal/locally produced datasets and/or non-HEASARC supported missions.

Documentation on how to install a local Caldb is covered in OGIP Calibration Memo "How to Install a Calibration Database" CAL/GEN/94-004.

NOTE: !!!!!!!!! THIS DOCUMENT IS IN PROGRESS !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Contents

1  INTRODUCTION
    1.1  Overview of Management using Perl Script
2  Diskspace Requirements for HEASARC-supported Missions
3  MANAGING HEASARC-SUPPORTED MISSIONS - exact copies
    3.1  Using Perl Script
    3.2  NOT Using Perl Script
4  MANAGING HEASARC-SUPPORTED MISSIONS - but with personal/locally produced datasets added
    4.1  Inclusion of New Datasets from the HEASARC (Using Perl Script)
    4.2  Inclusion of New Datasets from the HEASARC (NOT Using Perl Script)
    4.3  Inclusion of New Personal/Local Datasets
5  MANAGING NON-HEASARC-SUPPORTED MISSIONS and/or INSTRUMENTS
6  How do I get FTOOLS?
7  IF ALL ELSE FAILS ...
8  APPENDIX I
9  APPENDIX II

1  INTRODUCTION

This document is intended to provide instructions for `local Caldb managers' wishing to manage their local Calibration database (Caldb). What is meant by managing? This basically means that (periodically) when new calibration files are made available, the `local Caldb manager' needs to include these new files in their local Caldb and to update the associated caldb.indx files such that the new calibration files are indexed (and thus made accessible to the software). This is obviously necessary, whether the mission is HEASARC-supported or non-HEASARC-supported. It is expected however, that the great majority of sites/institutions will possess Caldb set-ups containing only missions/instruments which are HEASARC-supported.

At the HEASARC, scripts automatically produce compressed tar files of all the good quality calibration files and the appropriate caldb.indx file for a given mission/instrument (i.e., goodfiles_asca_sis.tar.Z for the asca sis). Upon the delivery of a new calibration file to the HEASARC Caldb by the various instrument teams, the latest calibration file and the updated caldb.indx file is automatically included in the HEASARC compressed tar file. It is then obviously recommended that the `local Caldb Manager' updates their local Caldb as soon as possible (by obtaining this new compressed tar file from the HEASARC, either by perl script or `by hand', see Sections 3 and 4). For information as to how you, the astronomical community, are notified of new calibration file deliveries to the HEASARC, please see Appendix II.

It is assumed that the FTOOLS software package is already available at your site since you presumably already possess a Caldb which has been successfully installed and is operable. The FTOOLS package of course, contains the installation, management & access software for your local Caldb, and your Caldb is of little use without it. Details of the latest version of FTOOLS, are in Section 6.

If your Caldb contains only missions/instruments which are exact copies of the HEASARC, and you wish to manage your Caldb in the simplest and easiest way, you can use the Perl script (manage_caldb_hcopy.perl) available from ???? (SCRIPT IN PROGRESS). Since the script uses file transfer protocol to download files from the HEASARC, your system manager will also need to install the package libnet-1.05_01.tar.gz from the Comprehensive Perl Archive Network (CPAN).
See http://www.perl.com/CPAN.
This package requires that Perl 5.002 or better be installed on your machine (Perl 5.002 however, does not contain libnet-1.05_01.tar.gz as part of its default distribution). For additional instructions please see APPENDIX I.

If your Caldb contains missions/instruments which are HEASARC-supported, but personal/locally produced datasets have been added, to manage these missions you will require FTOOLS from the !NEXT! FTOOLS release version 4.1. Furthermore, a perl script (manage_caldb_local.perl) is available from ????..SCRIPT IN PROGRESS.... which automates the management process (see above paragraph for installation procedures).

Advice on the management of non-HEASARC-supported missions/instruments is detailed in Section 5.

All questions, comments and suggestions should be directed to:
caldbhelp@athena.gsfc.nasa.gov
or to a member of Team Caldb listed at the URL:
/docs/heasarc/caldb/caldb_team.html

Team Caldb

A postscript file of the latest version of this document can be obtained from the anonymous ftp account on: legacy.gsfc.nasa.gov as
/caldb/docs/manuals/management/management_guide.ps

1.1  Overview of Management using Perl Script

and/or

2  Diskspace Requirements for HEASARC-supported Missions

At the HEASARC, scripts are running which automatically produce :

Such information is held on :
goodfiles_miss_inst_size.new and
goodfiles_miss_inst.tar.Z

available from the HEASARC via either the WWW at the URL:
/docs/heasarc/caldb/caldb_goodsize.html
the WWW or
anonymous ftp from heasarc.gsfc.nasa.gov in the directory /caldb/data/miss/inst
i.e. /caldb/data/asca/gis anonymous ftp (from within the appropriate sub-directory)

The mission/instruments currently available at the HEASARC are:

TABLE I

Mission Instrument Approx diskspace requirement
(Mbytes)
ASCA GIS 98
SIS 85
XRT 4
ROSAT HRI 1
PSPC 11
XRT 0.2
XTE PCA 14

However for the most up-to-date diskspace requirements please examine:
goodfiles_miss_inst_size.new

3  MANAGING HEASARC-SUPPORTED MISSIONS - exact copies

3.1  Using Perl Script

N.B. If you don't have Perl on your machine, please see Appendix I.

The script will prompt you for the following information :

  1. The name of the top-level directory to your Caldb i.e. /abc/def/caldbtop

  2. Confirmation that the full path given to your Caldb i.e. /abc/def/caldbtop is the correct one
    - If YES the program will continue

    - If NO the program will terminate

  3. The mission and instruments for which you wish to update calibration information and/or those which you now wish to include in your Caldb.

Self-explanatory messages are issued to the screen to indicate the success/failure of each stage of your Caldb update.

What happens if manage_caldb_hcopy.perl informs that:
``Some requested *.tar.Z files have NOT been successfully un-tarred ... please see /abc/def/caldbtop/caldb_err.log."

This message will be undoubtedly due to disk space problems. Simply
(i) Consult your systems manager regarding the aquisition of more diskspace.
(ii) Examine /abc/def/caldbtop/caldb_err.log to determine which mission/instrument(s) failed.
i.e. goodfiles_rosat_pspc.tar.Z
(iii) Run manage_caldb_hcopy.perl again, requesting the tar.Z file for the appropriate mission and instrument
i.e., rosat pspc.

What does the manage_caldb_hcopy.perl script actually do?

(i) Checks the full path given to your Caldb i.e. dirtop = /abc/def/caldbtop already exists. Also checks that your Caldb actually exists.
(ii) Changes the current working directory to dirtop
(iii) Obtains compressed tar files for the selected mission and instruments via anonymous ftp from heasarc.gsfc.nasa.gov.
(iv) Uncompresses and untars the files.
(v) Creates the appropriate directory structures (if necessary)
(vi) Populates the directory tree with calibration files
(vii) If the calibration information for a given mission/instrument is being updated, the local copy of the caldb.indx is replaced with the HEASARC copy. If the mission/instrument is being included in the Caldb for the first time the HEASARC copy of the caldb.indx is placed in /dirtop/data/missn/instr.

What happens if Perl version 5.002 or better is not installed on my system?

Unfortunately you will have to resort to the more long-winded installation method given below :

3.2  NOT Using Perl Script

  1. Set the current working directory to the top-level directory to your Caldb i.e. /abc/def/caldbtop

  2. Obtain the compressed tar files goodfiles_miss_inst.tar.Z for the mission and instruments in which you are interested :
    i.e. goodfiles_asca_gis.tar.Z

    available from the HEASARC via either the WWW at the URL:
    /docs/heasarc/caldb/caldb_goodsize.html
    the WWW or
    anonymous ftp from heasarc.gsfc.nasa.gov in the directory /caldb/data/miss/inst
    i.e. /caldb/data/asca/gis

    *** Then for each compressed tar file *** :

  3. Uncompress the file (i.e. uncompress *.tar.Z)

  4. Untar the file (i.e. tar xvf *.tar)
    - thus creating the same directory structure as at HEASARC Calibration Database and populating your local Caldb file-system.

  5. Delete the tar file

  6. You must index the calibration data files for the given mission/instrument :
    i.e. if the tar file was goodfiles_asca_gis.tar then
    mv the file data/tarredcif to data/asca/gis/caldb.indx.
    Thus replacing your local copy of data/asca/gis/caldb.indx with the new HEASARC copy.

4  MANAGING HEASARC-SUPPORTED MISSIONS - but with personal/locally produced datasets added

If you are updating this mission in your Caldb for the first time : whether you are including new calibration files from the HEASARC or new calibration files produced locally, you must intially

- run the FTOOL cifcadd (to add an extra column, CAL_ORIGIN, to the caldb.indx file, whilst retaining all the caldb.indx information therein).

- run the FTOOL caldbflag (to flag all the locally produced calibration files as `LOCAL' in the CAL_ORIGIN column of the caldb.indx file)

4.1  Inclusion of New Datasets from the HEASARC (Using Perl Script)

N.B. If you don't have Perl on your machine, please see Appendix I.

The script will prompt you for the following information :

  1. The name of the top-level directory to your Caldb i.e. /abc/def/caldbtop

  2. Confirmation that the full path given to your Caldb i.e. /abc/def/caldbtop is the correct one
    - If YES the program will continue

    - If NO the program will terminate

  3. The mission and instruments for which you wish to update calibration information.

Self-explanatory messages are issued to the screen to indicate the success/failure of each stage of your Caldb update.

What happens if manage_caldb_local.perl informs that:
``Some requested *.tar.Z files have NOT been successfully un-tarred ... please see /abc/def/caldbtop/caldb_err.log."

This message will be undoubtedly due to disk space problems. Simply
(i) Consult your systems manager regarding the aquisition of more diskspace.
(ii) Examine /abc/def/caldbtop/caldb_err.log to determine which mission/instrument(s) failed.
i.e. goodfiles_rosat_pspc.tar.Z
(iii) Run manage_caldb_local.perl again, requesting the tar.Z file for the appropriate mission and instrument
i.e., rosat pspc.

What does the manage_caldb_local.perl script actually do?

(i) Checks the full path given to your Caldb i.e. dirtop = /abc/def/caldbtop already exists. Also checks that your Caldb actually exists.
(ii) Changes the current working directory to dirtop
(iii) Obtains compressed tar files for the selected mission and instruments via anonymous ftp from heasarc.gsfc.nasa.gov.
(iv) Uncompresses and untars the files.
(v) Creates the appropriate directory structures (if necessary)
(vi) Populates the directory tree with calibration files
(vii) Updates the local copy of the caldb.indx for a given mission/instrument using the FTOOL mudcif (Many Updates to a Caldb.Indx File). mudcif indexes the (new) calibration files provided by the HEASARC appropriately, and flags the older versions of the new datasets as `bad' (whilst ignoring personal/locally produced datasets, hence ensuring these datasets are still accessible to the s/w).

What happens if Perl version 5.002 or better is not installed on my system?

Unfortunately you will have to resort to the more long-winded installation method given below :

4.2  Inclusion of New Datasets from the HEASARC (NOT Using Perl Script)

  1. Set the current working directory to the top-level directory to your Caldb i.e. /abc/def/caldbtop

  2. Obtain the compressed tar files goodfiles_miss_inst.tar.Z for the mission and instruments in which you are interested :
    i.e. goodfiles_asca_gis.tar.Z

    available from the HEASARC via either the WWW at the URL:
    /docs/heasarc/caldb/caldb_goodsize.html
    the WWW or
    anonymous ftp from heasarc.gsfc.nasa.gov in the directory /caldb/data/miss/inst
    i.e. /caldb/data/asca/gis

    *** Then for each compressed tar file *** :

  3. Uncompress the file (i.e. uncompress *.tar.Z)

  4. Untar the file (i.e. tar xvf *.tar)
    - thus creating the same directory structure as at HEASARC Calibration Database and populating your local Caldb file-system.

  5. Delete the tar file

  6. You must index the calibration data files for the given mission/instrument :
    i.e. run the FTOOL mudcif (Many Updates to a Caldb.Indx File), with the file
    /abc/def/caldbtop/data/input.ASCII as input. mudcif indexes the (new) calibration files provided by the HEASARC appropriately, and flags the older versions of the new datasets as `bad' (whilst ignoring personal/locally produced datasets, hence ensuring these datasets are still accessible to the s/w).

4.3  Inclusion of New Personal/Local Datasets

  1. Populate the appropriate sub-directories with your new local/personal calibration files.
    i.e. /caldb_topdirectory/data/xte/pca/cpf/responses/95apr03/local.rmf

  2. Index your cal files in the caldb.indx i.e. /caldb_topdirectory/data/xte/pca/caldb.indx by running the FTOOL udcif. Run udcif from the sub-directory which contains the cal file since the tool looks at the cwd to determine the path to the file and then writes the path to the CAL_DIR column in the caldb.indx and the filename to the CAL_FILE column.
    - input calibration filename
    - input name (& location) of the index file
    - input quality value of dataset being entered (0 ..for 'good' quality hopefully!)

  3. Run caldbflag (to flag all the new locally produced calibration files as `LOCAL' in the CAL_ORIGIN column of the caldb.indx file).

  4. Perform an fdump on the caldb.indx to check the files have been indexed correctly

5  MANAGING NON-HEASARC-SUPPORTED MISSIONS and/or INSTRUMENTS

Upon delivery of a new calibration file by the (non-HEASARC) mission/instrument teams to the `local' Caldb, you will need to intially examine the file to check that it conforms to OGIP-approved standards, and then (if correct), index the file into your Caldb. Overall, you will need to perform :

  1. fverify non_heasarc_stuff.rmf (to check that the header and data in the primary array and in every extension conform to the FITS format standard).

  2. fchecksum non_heasarc_stuff.rmf (to verify or update the CHECKSUM and DATASUM keywords in each HDU of the input FITS file).

    *** If the above FTOOLS reveal problems with the file, you should contact the instrument team who should then rectify them as necessary. Once the file is checked as `correct', you may continue with the following : ***

  3. Move non_heasarc_stuff.rmf to the correct Caldb sub-directory
    i.e. mv non_heasarc_stuff.rmf /caldbtop_directory/data/mission/instr/matrices

  4. Index non_heasarc_stuff.rmf into the appropriate caldb.indx file
    i.e. /caldbtop_directory/data/mission/instr/caldb.indx by running the FTOOL udcif. Run udcif from the sub-directory which contains the cal file since the tool looks at the cwd to determine the path to the file and then writes the path to the CAL_DIR column in the caldb.indx and the filename to the CAL_FILE column.
    - input filename
    - input name (& location) of the index file to be updated
    - input quality value of dataset being entered (0 ..for 'good' quality hopefully!)

  5. Perform an fdump on the caldb.indx to check the files have been indexed correctly

6  How do I get FTOOLS?

The latest FTOOLS software package is available from the HEASARC via either the WWW at the URL:
/docs/software/ftools/ftools_release.html
the WWW or anonymous ftp from heasarc.gsfc.nasa.gov in the directory
software/ftools/release
anonymous ftp Both areas contain the necessary release notes and the installation instructions.

7  IF ALL ELSE FAILS ...

Send e-mail to:
caldbhelp@athena.gsfc.nasa.gov
or contact a member of Team Caldb listed at the URL:
/docs/heasarc/caldb/caldb_team.html

Team Caldb

8  APPENDIX I

Since the Caldb management scripts manage_caldb_*.perl use file transfer protocol to download files from the HEASARC, your system manager will need to install the package libnet-1.05_01.tar.gz from the Comprehensive Perl Archive Network (CPAN). See http://www.perl.com/CPAN.
This package requires that Perl 5.002 or better be installed on your machine (Perl 5.002 however, does not contain libnet-1.05_01.tar.gz as part of its default distribution).

Your system manager should then replace the 1st line of the perl script

#!/usr1/local/bin/perl5

with the location of perl on your system. For example if perl is located in /usr/local/bin/perl then the 1st line should of the perl script should read

#!/usr/local/bin/perl

9  APPENDIX II

You will be notified that new calibration files are available through an e-mail distribution list. Three such distribution lists are currently available for the ASCA, ROSAT and XTE missions. They are called ascacaldb, rosatcaldb and xtecaldb respectively. You can subscribe to these lists by sending a one-line e-mail message to listserv@athena.gsfc.nasa.gov. The body of the message should be:

subscribe <list> <your name>

where <list> is either ascacaldb or rosatcaldb or xtecaldb and <your name> is your surname.

also on the web !!

USEFUL LINKS TO OTHER HTML PAGES

The following useful links are available (in the HTML version of this document only):



File translated from TEX by TTH, version 3.13.
On 4 May 2004, 12:14.