FTOOLS and HEADAS Filename Syntax

All of the ftools and HEADAS tasks support CFITSIO's extended file name syntax when specifying the names of input or output data files. The first section of this document describes the features of the syntax when specifying the name of a new data file that is to be created by a task, and the second section describes the features of the file syntax that apply when opening an existing data file.

A. WHEN CREATING A NEW FILE

1. Overwriting, or 'Clobbering' an Existing File

Most tasks will exit with an error if asked to create a new file that already exists, unless one of the following conditions is met:

If the name of file is preceded by an exclamation point (!) then any existing file with that same name will be deleted before the new empty file is created. Note that the exclamation point must be preceded with the backslash (\) escape character when it is used on the Unix command line, as in this example:
ftcopy infile.fits \!outfile.fits
Most tasks have a 'clobber' hidden parameter which, if set equal to 'Yes' when the task is run, will cause any existing file to be deleted before the new file is created.

2. Compressed Output Files

To save disk space, the output data file may be written in a compressed format. Two different types of compression algorithms are supported:

If the specified output file name ends with the suffix '.gz' then the file will be compressed using the same algorithm as used by the Unix 'gzip' program. Typical files can be compressed by a factor of 2 or 3 using this technique, but the actual compression factor depends greatly on the contents and datatype of the arrays in the file. CFITSIO supports this compression feature by first creating an uncompressed version of the file in memory, which is then compressed and written to disk when the application program closes the file.
If the specified output file name ends with the 'compress' qualifier contained within square brackets (e.g. 'outfile.fits[compress]') then the primary array image and any image extensions in the file will be compressed using the recently developed 'tile' compression algorithm. In this format the image is divided into a rectangular grid of tiles and each tile is then compressed and stored in a row of a FITS binary table. The header keywords remain uncompressed for easy and rapid access by tasks that later read the file. For more information about the tile compression format, see the 'tilecompression' help file.

3. Using a Template to Create a New FITS File

The structure of any new FITS file that is to be created may be defined in an ASCII template file. If the name of the template file is appended to the name of the FITS file itself, enclosed in parenthesis (e.g., 'newfile.fits(template.txt)' ) then CFITSIO will create a FITS file with that structure before opening it for the application to use. The template file basically defines the dimensions and data type of the primary array and any IMAGE extensions, and the names and data types of the columns in any ASCII or binary table extensions. The template file can also be used to define any optional keywords that should be written in any of the HDU headers. The image pixel values and table entry values are all initialized to zero. The application program can then write actual data into the HDUs. See the 'template' help file for a complete description of the template file syntax.

Applications programs usually must be specifically designed to support the use templates when creating new FITS files. Most of the existing ftools and HEADAS tasks do not support templates. One exception, however, is the fttemplate (or ftemplate) task which uses a template to create a new file.

4. Creating a Temporary Scratch File in Memory

It is sometimes useful to create a temporary output file when testing an application program. If the name of the file to be created is specified as "mem:" or "mem://" then CFITSIO will create the file in memory where it will persist only until the program closes the file. Use of this "mem:" output file usually enables the program to run faster, and of course the output file does not use up any disk space.

B. WHEN OPENING AN EXISTING FILE

1. Basic Types of Input Data Files

The following types of data files may be opened by any ftools or HEADAS task:

Compressed files with .zip, .gz (gzip), or .Z (Unix compress) filename extensions. It is not necessary to include the compression suffix when specifying the name of the input file. When opening compressed files, CFITSIO first uncompresses the entire file into memory, and the applications program then reads from that file in memory. Compressed files can only be opened with 'readonly', not 'readwrite' file access.
FITS files on the internet can be opened by specifying the full FTP or HTTP URL to the file. These files can only be opened with 'readonly' file access.
FITS files can be piped between tasks on the stdin or stdout file streams. The filenames 'stdin', 'stdout', or '-' are reserved for this purpose. For example, if Task1 writes an output FITS file, and Task2 reads an input FITS file, then the output of the first task may be piped to the other task with either of the following commands:
task1 stdout | task2 stdin task1 - | task2 -
IRAF format images (with a .imh file extension) may be opened with 'readonly' access. The IRAF file is converted on the fly into a virtual FITS file in memory, which is then opened and read by the application program.
Data arrays in raw binary data format may be opened as pseudo FITS images with 'readonly' access. CFITSIO converts the data into a FITS image in memory on the the fly which the application program then reads. The datatype and dimensions of the image are specified within square brackets following the root name of the binary data file, as in 'rawfile.dat[iB300,200]'. The first letter within the brackets specifies the data type, where,
An optional second character specifies the byte order of the array, where B = big endian (as on a SUN Unix machine) or L = little endian (as on an IBM PC). If the second character is omitted then the native byte order of the local machine is assumed. This is followed by a comma-separated list of integers giving the size of the dimensions in the image array, where the most rapidly varying dimension is given first (as in Fortran array declarations and opposite to the order of multidimensional array declarations in C). This may be followed by a colon (:) and another integer value which gives the number of bytes to be skipped at the beginning of the file before the start of the actual image pixel data. Examples of this raw file specification are:
datafile1.dat[i300,200] - A 300 x 200 short integer image.
datafile2.dat[r512,512:128] - A 512 x 512 real image. The first 128 bytes in the file will be ignored.
FITS files can be read and written in shared memory. This can potentially achieve much better data I/O speed compared to reading and writing the same FITS files on magnetic disk. Currently, a maximum of 16 files may be created in shared memory, with names of the form 'shmem://h0' to shmem://h15'.

2. Extension Specifier

In many cases the ftools or HEADAS application program must be told which particular extension (also known as 'HDU') within the input data file that should be opened and operated on. This is done by specifying the extension name (as given by the EXTNAME or HDUNAME header keyword) or the extension number (where the primary array is number 0) in square brackets following the file name (e.g., 'myfile.fits[events]' means open the 'EVENTS' extension, or 'myfile.fits[1]' means open the first extension following the primary array in the file). Alternatively, one may specify the extension number following a 'plus sign' (e.g., 'myfile.fits+1'). One can also specify the 'version' number of the desired extension and the type of extension (Image, Binary table, or ASCII table) to uniquely identify the HDU, if more than on HDU in the file has the same EXTNAME value (e.g., 'myfile.fits[events,2,binary]'). The extension type my be abbreviated with the letters 'i', 'b', or 'a'.

If an explicit extension name or number is not specified as part of the input file name then many of the newer ftools and HEADAS tools will automatically attempt to open the first 'interesting' table or image in the FITS file. If the application is expecting to open a FITS table, then it will move to the first table extension in the file that is not a Good Time Interval (GTI) table. If the application is expecting to open an image, it will open the first image HDU that it finds that is not a null image (with zero dimensions). Some of the older ftools tasks were not designed to use this feature, and hence will simply open the primary array in the input file if no extension is specified.

3. Virtual File Filtering

A variety of different types of filtering operations may be specified when opening a file. CFITSIO performs the filtering operations and creates a virtual file (usually in memory) which the application program then reads. The filtering specifications are appended to the root file name and enclosed in square brackets. These filters provide very powerful data manipulation and editing capabilities to all of the ftools and HEADAS tasks.

The virtual file filters are described in the following separate help files:

imfilter
- select a subimage out of a larger image
- open a N-dimensional vector in a binary table element as if it were a primary array image
pixfilter
- apply mathematical operations to the input image pixel values
binfilter
- create an image histogram by binning table columns
colfilter
- create or modify header keywords and table columns
rowfilter
- select a subset of rows from a table based on a boolean expression. This includes filtering on Good Time Interval extensions and Spatial Region files.
calc_express
- gives the detailed syntax for row filters

EXAMPLES

When creating a new output file:

!myfile.fits - the leading exclamation character indicates that any existing file with that name should be deleted prior to creating the new file.
\!myfile.fits - when the clobber flag is specified on the Unix command line, the '!' character must be preceded by a backslash escape character.
myfile.fits.gz - the output file will be compressed using the gzip compression algorithm when the file is closed and written to disk
'myfile.fits[compress]' - any images that are written to the file will be compressed using the new 'tile compression' technique.
'myfile.fits(template.txt)' - the new file will be created having the structure that is defined in the ASCII template file.
'mem://': creates a scratch output file in core computer memory. The resulting 'file' will disappear when the program exits, so this is mainly useful for testing purposes when one does not want a permanent copy of the output file.

When opening an existing file:

myfile.fits - the simplest case of a FITS file on disk in the current directory.
'myfile.fits[3]' - open the 3rd HDU following the primary array. The name must be enclosed in single or double quote characters when using square brackets.
myfile.fits+3 - same as above
'myfile.fits[EVENTS]' - open the extension that has EXTNAME = 'EVENTS'
'myfile.fits[EVENTS, 2]' - same as above, but also requires EXTVER = 2
'myfile.fits[events,2,b]' - same, but also requires XTENSION = 'BINTABLE'
'myfile.fits.gz[events, 2]' - opens and uncompresses the file myfile.fits then moves to the extension which has the keywords EXTNAME = 'EVENTS' and EXTVER = 2.
'-' - a dash (minus sign) signifies that the input file is to be read from the stdin file stream, or that the output file is to be written to the stdout stream.
'ftp://heasarc.gsfc.nasa.gov/test/vela.fits' - FITS files on any ftp server on the internet may be directly opened with read-only access.
'http://heasarc.gsfc.nasa.gov/software/test.fits' - any valid URL to a FITS file on the Web may be opened with read-only access.
'shmem://h2[events]' - opens the FITS file in a shared memory segment and moves to the EVENTS extension.