1. Running the OSSE Low-Level Pipeline

1.1 Intent

This document describes procedures that the Archive Technician shall follow to process low-level OSSE data (spectral and pulsar) at the COSSC. These procedures will manipulate data from native format on the delivered tapes to compressed FITS format in the COSSC archive.

1.2 Outline

Low-level OSSE data are generally delivered one viewing period at a time on three separate 4mm DATs (see Section 1.3).

The procedure for processing low-level OSSE data is composed of four main segments which run sequentially. All work is done in the OSSEPIPE account on VMS and the ARCHIVER account on Unix.

1.

Restores data files in native VAX/VMS format from tape to disk.

2.

submits a batch job that converts the native format files to FITS files in a staging area. It also uses VMS TAR and VMS BACKUP to package up text and other files into a single file. The files are also compressed with VAX GZIP in this staging area. The staging area is NFS mounted between the VAX and Unix clusters - an aspect important for the operation of the third segment of this process. These first two segments run exclusively on VAX/VMS. Attempts to process them on the Alpha/VMS platforms ran into problems with the difference in numeric formats between the VAX and Alphas (see Section A.1).

3.

The next segment extracts the FITS keywords necessary for the W3Browse database and then transfers the data files to the archive. The IDL routine MRDFITS is used for the keyword extraction since this function can very quickly read the FITS keywords from the primary header of compressed files. This final step is run on the Unix systems and requires the input FITSified data be available on the Unix side (hence the NFS mounting in the previous step).

4.

The final segment generates the TDAT file used by the HEASARC database engine for accessing the archive.

It may be worthwhile in the future to incorporate the PDF file processing into the SDB pipeline.

For control purposes, a processing form is maintained for each data delivery and installation into the archive. For this portion of the pipeline, this form is called OSSEPIPE-01. You should utilize the most recent release of this form.

  

Figure 1.1: Diagram of the pipeline for OSSE low-level spectral data. In the archive path specification, {nn} is the phase number and {dddd_d} is the 6 character viewing period with the underscore replacing the decimal point.

   
1.3 Data Tapes

Data from the OSSE instrument are delivered by the Naval Research Laboratory (NRL) after the proprietary term expires. Typically, a delivery consists of three tapes containing OSSE data for a single viewing period. Each tape is labelled with a code composed of six characters. The first two characters can be:

OA	SDB (spectral) data
OB	PDF (timing & pulsar) data
OD	TELEMETRY data (which is not archived)

The last four characters of the label are digits denoting the first day of the viewing period. Specifically, the first digit in the code is the last digit of the year number, and the last three digits of the code are the day number within that year. For instance, a tape labeled ``OB2136'' contains PDF data whose viewing period began on the 136th day of 1992 (year number ends in 2).

Log listings of the contents of the delivered data tapes are stored in $DISK10:[INVENTORY.OSSE].

1.4 Preparing the Pipeline Run

For ISO-9000 and control purposes, each viewing period processed will be recorded on a pipeline control form. For the low-level spectral pipeline, the form is designated OSSEFORM-01.

When the data tapes arrive and are recorded in inventory, the header of the form should be filled out. The header records the basic inventory information - CGRO viewing period number, the data delivery number, tape identifier label, and the tape delivery date.

Get the viewing period date and length from the CGRO viewing plan.

The length of the proprietary period is determined by the CGRO phase number per the table below:

phase	length
$\leq 6$	12 months
7	6 months
8	3 months
$\geq 9$	none

From this, determine the date of the end of the proprietary period and record it on the form. If this date has passed, you may continue with the steps to load the data into the archive.

1.4.1 Notes

The total disk space required to process an entire delivery is substantial. In addition to the space required to restore the files from tape to disk in native format, the FITS conversion process will generate output files with an even larger space requirement. For low-level SDB files, for example, the output FITS files in the staging area will require approximately 1.6 times the space occupied by the input files. It is the responsibility of the Archive Technician to check that sufficient disk space exists before performing the processing, and to verify that the entire procedure has completed successfully afterwards.

1.5 Running the Data File Restoration Program

The program to restore data files from tape to disk assumes that the desired tape is loaded in tape drive connected to GROVX0 and referenced by the symbol OSSE_TAPE, and must be run from the OSSEPIPE account on GROVX0. The restoration software includes performing the allocation and mounting of the tape drive.

To execute, type:

@SDB_RESTORE  [deliv_num]  [tape_label]

or

@PDF_RESTORE  [deliv_num]  [tape_label]

where [deliv_num] is the tape delivery number and [tape_label] is the identifier marked on the tape.

For example, the command

@SDB_RESTORE 111 OA3361

will restore all files from tape OA3361 in delivery 111.

1.6 Running the FITS Conversion Program

The next step is to convert the data files from native format to FITS format. This is accomplished by submitting the format conversion program to a batch queue from the OSSEPIPE account.

To execute, type:

@SDB_BATCH  [deliv_num]  [view_period]  [time]

or

@PDF_BATCH  [deliv_num]  [view_period]  [time]

where [view_period] is the corresponding viewing period number (note there should always be one digit to the right of the decimal point) and [time] is an (optional) start time for the batch job [hh:mm]

For example, the command

@SDB_BATCH 111 313.0 20:00

will submit the FITS conversion software as a batch job that will begin at 20:00 to process the SDB files from delivery 111, containing data obtained during viewing period 313.0.

1.6.0.1 Special Case of Two Viewing Periods in One Delivery

As noted in Section 1.3 above, a data delivery from NRL typically covers one viewing period. On occasion, however, a delivery may contain data from two viewing periods. No indication exists on the tapes themselves, however, as to the demarcation date between the two viewing periods. For such cases, the Archive Technician must refer to the CGRO observation timelines to determine this date. Then, a subdirectory appropriately numbered for the second viewing period should be manually created in the staging area, and the corresponding FITS files moved into that directory. A one-day overlap of data files should be maintained between the two viewing periods.

1.7 Final Installation into the Archive

On completion of FITSification, the pipeline activity moves over to the Unix side of the system in the ARCHIVER account. Due to the file naming convention for the resulting TDAT files, this step can be run only once per day. Also note that case is important when typing in these commands.

Once in Unix

>cd osse
>idl

whichs sets up the account to run IDL v5.0 and starts IDL. Once in IDL, we can transfer the files to the archive:

.run tdat_library
.run archive_osse

The archive_osse process builds the output directory structure and transfers the data files to the archive. It will generally run for a while (an hour or so). The final step is

.run build_new_tdat

which examines the FITS headers and extracts the information to generate the data tags for the entire OSSE archive. This data is tabulated in a pipe-delimited file: osse_nnnnnnn_build.tdat which appears in the osse_tdat directory on completion of this program (nnnnnnn is the Julian Day number). A copy of this file is also sent (via FTP) to hrcdba@dbsrv:/dba_dbase/work/osse/hdb/. On completion, an e-mail notification is sent to dba_browse@lheamail.gsfc.nasa.gov. When this process completes, type

exit

to leave IDL.

1.8 Clean-up Operations

As a precaution, the pipelines perform only a limited amount of `clean-up' behind them to facilitate restarting in the event a run fails. The actual clean-up is best performed manually after there has been plenty of time to ensure the previous runs did complete successfully.

There are two subdirectory areas that should be empty before initiating a new pipeline run. WARNING - MAKE SURE THAT ALL OSSE PIPELINE PROCESSES ARE COMPLETE BEFORE RUNNING THESE PROGRAMS.

• The directories below $DISK20:[OSSEPIPE.NATIVE] should be gone. This is the location were data is placed when restored from the delivered data tapes. This can be done with the command:

  @PURGE_NATIVE [dn]
  

  where [dn] is either the data delivery number, to delete a specific delivery, or `*' to delete all low-level data deliveries from the native area. The program prompts you with a warning of what it is about to do. Nonetheless, use it with caution.

• The directories $DISK20:[OSSEPIPE.STAGE.PHASE*...] should all be emptied and deleted. This is the staging area where the FITSified data files are placed after creation and from where they are retrieved for compression and copying into the actual archive. This can be performed with a single command:

  @PURGE_STAGE [type] [vp]
  

  where [type] is the data types: lolevspec, hilevspec, pulsar, or `*' to purge all types; and [vp] is the viewing period you desire to delete, or `*' to delete all.