NOTICE: This Legacy journal article was published in Volume 3, May 1993, and has not been updated since publication. Please use the search facility above to find regularly-updated information about this topic elsewhere on the HEASARC site.

FITS Software Update

The first issue of Legacy contained an article describing the FITSIO and FTOOLS software at the HEASARC. There have been significant new developments to this software since then, so this article provides an update on the FITS software status.

FITSIO Status

FITSIO is a machine-independent Fortran-77 subroutine interface for reading or writing data files in FITS format. It provides an easy-to-use interface which insulates the programmer from having to deal directly with the complicated formatting details of a FITS file. FITSIO supports all the currently defined types of FITS files and runs on most popular computers. Version 3.31 of FITSIO was released in February 1993, and the next release is planned for May 1993.

There are currently two areas where significant new features are being added to FITSIO. One is that a C programming interface, called FITSIOC, has been provided by HEASARC programmer Bruce O'Neel. FITSIOC consists of a thin layer of C interface routines which sit on top of the Fortran FITSIO code and handle all the system-dependent details required when calling a Fortran subroutine from a C program. Functionally, FITSIOC makes it appear to a C programmer as though FITSIO were written in C. This makes it possible to call FITSIO from C programs in a more convenient and machine-independent way. FITSIOC is currently undergoing beta testing and should be generally available with the next release of FITSIO.

The second main area of enhancement to FITSIO is to add data compression capabilities to the interface. There has been a lot of discussion in FITS interest groups about the need for data compression in FITS files, and how best to do it; the current work within FITSIO should be considered as an experimental prototype useful for testing some of these concepts.

The first data compression scheme that has been added to FITSIO uses the IRAF `Pixel List' algorithm which compresses the file by looking for adjacent pixels in the image which have the same value. This is only useful for limited types of images, but works very well for typical X-ray images which usually contain many zero-value background pixels. Of more interest than the particular compression algorithm used here is the new general image interface that is being provided in FITSIO. This new set of interface routines access the image on a row by row basis and can read or write normal uncompressed FITS images as well as images that have been compressed using a variety of different algorithms. Currently, only a single compression algorithm is supported, but any number of other algorithms could be added. The decompression of the images is handled transparently by FITSIO, so the application code does not have to deal with the particular compression algorithm details.

This support for FITS data compression schemes will be provided on an experimental basis with the next release of FITSIO. Potential users of this feature should be warned, however, that it may undergo significant modification before being fully supported.

FTOOLS Status

FTOOLS is a collection of utility programs to create, examine, or modify the contents of FITS data files (and was first described in Legacy 1, 16). While FTOOLS is designed to support all types of FITS files, it is especially geared toward processing data in the FITS Binary Table format which provides a very flexible way of efficiently storing almost any type of data and is being widely used by the HEASARC for high energy data files. A binary table consists of rows and columns of information where all the data in a column has the same datatype (e.g., bit, byte, integer, floating point, character string, complex). There may be an arbitrary number of columns in a table, each with a different data type. Each table element (i.e., the intersection of a row and column) may be a simple scalar number or may contain a whole vector of numbers. In simple binary tables every row of the table has the same length, but, in general, one can also create variable length tables where the size of vector table elements varies from row to row in the table. Finally, in addition to the table data itself, the binary table can contain any arbitrary number of header records which provide descriptive data appropriate either to the table as a whole or to individual columns of the table.

Version 2.3 of the FTOOLS software was relased in May 1993. The FTOOLS package is still evolving rapidly so new releases of the software are planned at approximately monthly intervals. The FTOOLS package has been written by a small group of programmers and has grown over the past year to include more than 45 individual tasks. Each task is a separate program which may be run in the IRAF environment or standalone on either a unix or VMS operating platform. All the FTOOLS programs share several common design features:

• all are written in ANSI Fortran or C;

• all use a simple standardized subroutine interface for getting the value of program parameters (e.g., the name of the input FITS file), and;

• all data I/O is restricted to FITS files via the FITSIO subroutine interface, or to simple ASCII format files in certain circumstances.

Most of the initial set of FTOOLS tasks perform very generic functions which could be appropriate for any type of FITS file. These tasks do things like list the contents of a FITS file or extract certain information from one FITS file and write it to another FITS file. Beyond this basic set of utility tasks, there are a number of mission specific FTOOLS tasks which perform calibration or analysis functions on various kinds of high energy astrophysics data. Since much of this project was funded by the Astro-D mission (now called ASCA), many of the FTOOLS tasks perform specific kinds of processing or calibration of the ASCA data. The following list gives a brief description of the currently available FTOOLS tasks.

General Purpose Utilities:

        flcol - List column information in a FITS table extension
      fstruct - List a description of the structure of a FITS file
        fdump - Dump contents of a FITS table to an ASCII file
      fimgdmp - Dump contents of a FITS image to an ASCII file
     ffilecat - Copies keyword values from a list of FITS file to FITS Table
     fcatdiff - Compares columns of a fits file and reports row differences
      fcreate - Create a FITS table from  ASCII input files
      fappend - Append a FITS extension onto another FITS file
     fextract - Copy a FITS extension from a file into a new file
       fmerge - Merge rows from several FITS tables into one FITS table
     fproject - Copy specified columns of a FITS table to a new table
      fselect - Create a new table from selected rows of a table
        fsaoi - Translate an SAOImage region file to an input file for fselect
        fsort - Sort a FITS table in place
     fmemsort - Fast memory sort of a FITS table
        fcalc - Calculates values for a column using an arithmetic expression

   fstatistic - Calculate mean, standard deviation, min, and max for a column
       farith - Perform arithmetic on 2 FITS images
      fcarith - Preform arithmetic on FITS image with a constant
       fhisto - Make a histogram of a column in a table
       fcurve - Make a light curve histogram from a column in a table
     f2dhisto - Make a 2-D histogram from 2 columns in a table
      fim2lst - Convert a 2D image to a pixel list (inverse of f2dhisto)
      fmrgmsk - Merge 2 or more spatial masks
    fmaskfilt - Filter an event list based on an input mask image
       findex - Create an index file for a FITS table column

      fparkey - Copy a parameter value to a FITS header keyword
      fkeypar - Copy a FITS header keyword to a parameter
      fpartab - Copy a parameter value to a FITS table element
      ftabpar - Copy a FITS table element to a parameter value
      ftabkey - Copy a FITS table element to a FITS header keyword
      fkeytab - Copy a FITS header keyword to a FITS table element
     fmodhead - Modify the header keywords in a FITS file
    fkeyprint - Display keyword(s) in FITS header(s)
       fburst - Remove bursts of events from time ordered event list
        fplot - Plot columns in a FITS file using QDP/PLT package


High Energy Astrophysics specific tasks:

     maketime - Calculate time intervals (GTIs) from housekeeping (HK) data
       mgtime - Merge 2 or more time interval (GTI) files
       fltime - Filter an event list within given time intervals (GTIs)
     hkexpand - Expand a compressed format housekeeping (HK) data file
   hkunexpand - Compress an expanded format houskeeping (HK) data file
        faint - Convert AstroD faint mode data to a bright mode format
     sec2time - Convert time offset to absolute time
     time2sec - Convert absolute time to a time offset
        coord - Convert (X,Y) pixel coordinates to (RA,Dec)
      hkscale - Scales a FITS houskeeping data file into physical values
       sisrmg - Generate SIS instrument response matrix
       gislin - Transform from ADC electronic units to physical coordinates
      ghkdump - Display GIS housekeeping parameters of the GIS HK file
       bldarf - Routine to create an xspect "arf" file
       grppha - Manipulates OGIP standard PHA FITS file
      gqaplot - GIS quick analysis ftool ***


Calibration specific tasks:


      st2rpsf - reads stw FITS file and writes to CALRPSF format file
       quzcif - interogates Caldb for location of a dataset
       dmprmf - Displays OGIP standard Response FITS file

     *** May not be supported at all sites

The FTOOLS package is currently available via anonymous ftp from heasarc.gsfc.nasa.gov in the /software/ftools/release subdirectory. This subdirectory contains compressed tar files for SUN/sparc, DEC/ultrix, and VAX/VMS distributions of the software. In addition there are several compressed postscript documents which cover the installation and usage of the FTOOLS package. We expect that FTOOLS will continue to grow rapidly, so updates to these distribution files will be made at frequent intervals.

ToFU Status

The ToFU (To Fits Utilities) library is a set of Fortran-callable subroutines designed to facilitate the direct conversion of data into FITS format. ToFU routines conceptually perform three operations: (1) create FITS headers, (2) write byte-streams to a FITS ASCII or binary table and (3) read the contents of a FITS ASCII or binary table and return it as a stream of bytes.

ToFU was originally created out of necessity by the Compton Observatory Science Support Center (COSSC) division of LHEA OGIP to deal with the task of writing a large number of data conversion programs to convert Compton Gamma Ray Observatory data into FITS format. However, ToFU's general treatment of FITS headers and input data makes it a potentially useful tool set for other astronomical FITS data conversion needs.

Two concepts, the template file and the byte-stream, are central to the way ToFU routines ingest data and build FITS files. Template files contain ASCII text instructions that tell ToFU routines how to populate the keywords and keyword values of a FITS header. The syntax of these instructions has been devised such that any valid FITS header constitutes a valid template file. Programs written with ToFU library routines read one or more of these template files (one template per desired FITS header) and construct the FITS file headers from the information contained within. ToFU template files make possible the modification of FITS headers and formats without the need for code modification and re-compilation. Template files also allow projects to maintain the integrity their FITS data formats since all ToFU-based programs on a given system can be directed to use template files from a configuration-managed directory.

Once the appropriate ToFU routine has read a template file and constructed a FITS header from it, another routine reads a specified data file and writes the contents to the FITS file. ToFU regards all data input files as byte-streams. This allows ToFU routines to treat any data file as an ordered set of bytes that are to be read in according to the information present in the last-written FITS header. For example, if a FITS binary table column has a TTYPE value of `3E', then ToFU reads 12 bytes from the input data file (input byte-stream) and writes them to the FITS file as three 4-byte real numbers. The byte-stream paradigm is the key to ToFU's ability to process a diverse number of data format types.

The ToFU library comes in three flavors: DECstation, VAX/VMS and SUN. Any of these flavors along with documentation may be obtained via anonymous ftp on heasarc.gsfc.nasa.gov in the compton/software/tofu directory. Each distribution contains a libmake utility that builds the ToFU library. To build and run ToFU requires a Fortran compiler, C compiler and version 3.2 or higher of the FITSIO library.

Proceed to the next article

Return to the previous article

Select another article