2. Running the OSSE High-Level Pipeline

2.1 Intent

This document describes procedures that the Archive Technician shall follow to process high-level OSSE spectral data at the COSSC.

2.2 Outline

The procedure for processing OSSE data is composed of three main segments that run sequentially. All work is done in the OSSEPIPE accounts on the VAX and Unix platforms.

1.

Restores data files in native format from tape to a VAX-mounted disk.

2.

Submits a batch job which FITS-ifies the SDB files and uses VMS TAR to package other support files. All files are then compressed using GZIP.

3.

The third segment runs under Unix and copies the FITSified data products from the staging area into the actual archive.

The complete pipeline is illustrated in Figure 2.1).

  

Figure 2.1: Diagram of the pipeline for OSSE high-level spectral data. In the archive path specification, {nn} is the phase number and {dddd_d} is the 6 character viewing period with the underscore replacing the decimal point. {target_name} is a version of the target name modified for compatibility with the file name syntax.

2.3 Data Tapes

Tape labels for high-level OSSE data products have ``OE'' in their label prefix.

2.4 Running the Data File Restoration Program

The high-level pipeline does not generate TDAT products for installation in the HEASARC database. Therefore, any high-level data products processed before the low-level data products will not be visible in W3Browse. In general, the low-level data products should be placed online before their high-level counterparts.

The high-level data products are placed online through a DCL script. The data tape is written in VMS BACKUP format.

1.

You must be logged in on the OSSEPIPE account on the node GROVX0 (GROVX0 is the only node which has access to the 4mm DAT).

2.

Mount the tape in the appropriate tape drive. From the keyboard, type the command

$ @HLP_RESTORE

This process will restore the backup set to disk in the area indicated in the DCL symbol OSSE_HLPNATIVE. When complete, the procedure will dismount the tape. Remove the tape from the drive and file it appropriately.

This process also performs the step of fixing a file name problem with the *BLN.SDB files. This is a bug which should only affect older deliveries. Basically, the portion of the BLN file prior to .SDB may be truncated due to an old 20-character filename limit in an earlier release of the IGORE package. This process actually occurred at NRL.

2.5 FITSification and Generating Other Data Products

This process requires IGORE (and IDL) so it must be run on a VAX. This is also run from the OSSEPIPE account but on the GROSSC machine.

Run the program from your root directory on VAX with

$ @HLP_BATCH [deliv_num]

where [deliv_num] corresponds to the high-level delivery number (without the letter). This procedure submits the job to the GROSSC_ONEJOB$BATCH queue. This queue has no time limit since this process may take a long time to run. This procedure assumes the High Level OSSE products are available in OSSE_HLPNATIVE. It consists of three major steps:

1.

FITSify all SDB files. Generates the XSPEC products if TOT files are in the data set. Wrap other supporting files and compress the entire set.

2.

Scan the High-Level directory and build/update the status file.

3.

Generate XSPEC files for the appropriate pointings. These are placed in the High-Level directories in OSSE_STAGE.

On completion of the FITSification, you should clean out the native high-level directory before performing another FITSification run.

The directories below $DISK20:[OSSEPIPE.NATIVE.HILEV] should be gone. This is the location were data is placed when restored from the delivered data tapes. This can be done with the by first enabling file and directory deletion with the command:

 SET PROTECTION=OWNER:RWED \$DISK20:[OSSEPIPE.NATIVE.HILEV\ldots]*.*

then delete them with with several applications of the command:

 DELETE \$DISK20:[OSSEPIPE.NATIVE.HILEV\ldots]*.*;*

Exercise extreme caution when performing this step. Currently, PURGE_NATIVE does not support deletion of the high-level data products.

2.6 Final Installation into the Archive

On completion of FITSification, the pipeline activity moves over to the Unix side of the system in the ARCHIVER account. Due to the file naming convention for the resulting TDAT files, this step can be run only once per day. Also note that case is important when typing in these commands.

Once in Unix

>cd osse
>idl

whichs sets up the account to run IDL v5.0 and starts IDL. Now we can transfer the files to the archive:

.run tdat_library
.run archive_osse

and when it completes, type

exit

Note that (unlike the low-level pipeline, you do not need to run build_new_tdat. This program will build the output directory structure in the archive and copy the files from the staging area to the archive directory.

2.7 Clean-up Operations

As a precaution, the pipelines perform only a limited amount of `clean-up' behind them to facilitate restarting in the event a run fails. The actual clean-up is best performed manually after there has been plenty of time to ensure the previous runs did complete successfully.

There are two subdirectory areas that should be empty before initiating a new pipeline run. WARNING - MAKE SURE THAT ALL OSSE PIPELINE PROCESSES ARE COMPLETE BEFORE RUNNING THESE PROGRAMS.

The directories $DISK20:[OSSEPIPE.STAGE.PHASE*...] should all be emptied and deleted. This is the staging area where the FITSified data files are placed after creation and from where they are retrieved for compression and copying into the actual archive. This can be performed with a single command:

 @PURGE_STAGE [type] [vp]

where [type] is the data types: lolevspec, hilevspec, pulsar, or `*' to purge all types; and [vp] is the viewing period you desire to delete, or `*' to delete all.