10.10 Column and Keyword Filtering Specification

The optional column/keyword filtering specifier is used to modify the column structure and/or the header keywords in the HDU that was selected with the previous HDU location specifier. This filtering specifier must be enclosed in square brackets and can be distinguished from a general row filter specifier (described below) by the fact that it begins with the string 'col ' and is not immediately followed by an equals sign. The original file is not changed by this filtering operation, and instead the modifications are made on a copy of the input FITS file (usually in memory), which also contains a copy of all the other HDUs in the file. (If a `#' character is appended to the name or number of the table HDU then only the primary array, and none of the other HDUs in the input file will be copied into memory). This temporary file is passed to the application program and will persist only until the file is closed or until the program exits, unless the outfile specifier (see above) is also supplied.

The column/keyword filter can be used to perform the following operations. More than one operation may be specified by separating them with commas or semi-colons.

The expression that is used when appending or recomputing columns or keywords can be arbitrarily complex and may be a function of other header keyword values and other columns (in the same row). The full syntax and available functions for the expression are described below in the row filter specification section.

If the expression contains both a list of columns to be included and columns to be deleted, then all the columns in the original table except the explicitly deleted columns will appear in the filtered table. If no columns to be deleted are specified, then only the columns that are explicitly listed will be included in the filtered output table. To include all the columns, add the '*' wildcard specifier at the end of the list, as shown in the examples.

For complex or commonly used operations, one can place the operations into an external text file and import it into the column filter using the syntax '[col @filename.txt]'. The operations can extend over multiple lines of the file, but multiple operations must still be separated by commas or semi-colons. Any lines in the external text file that begin with 2 slash characters ('//') will be ignored and may be used to add comments into the file.

When using column filtering to open a file “on the fly,” it is permitted to use multiple column filtering expressions. For example, the syntax

  filename.fits[col *][col -Y][col Z=X+1]
would be treated as equivalent to joining the expressions with semicolons, or
  filename.fits[col *; -Y;col Z=X+1]
Please note that if multiple column filtering expressions are used, it is not permitted to also use the [col @filename.txt] syntax in any of the individual expressions.

It is possible to use wildcard syntax to delete either keywords or columns that match a pattern. Recall that to delete either a keyword or a column, precede its name with a '-' character.

Wildcard patterns are: '*', which matches any string of characters; '?', which matches any single character; and '#' which matches any numerical string. For example these statements:

  -VEL*;       # remove single column (or keyword) beginning with VEL
  -VEL_?;      # remove single column (or keyword) VEL_? where ? is any character
  -#DEC_*;     # remove single keyword beginning with DEC_
  -#TUNIT#;    # remove single keyword TUNIT ending w. number
will remove the columns or keywords as noted. Be aware that if a '#' is not present, the CFITSIO engine will check for columns with the given name first, followed by keywords.

The above expressions will only delete the first item which matches the pattern. If following columns or keywords in the same CHDU match the pattern, they will not be deleted. To delete zero or more keywords that match the pattern, add a trailing '+'.

  -VEL*+;       # remove all columns (or keywords) beginning with VEL
  -VEL_?+;      # remove all columns (or keyword) VEL_? where ? is any character
  -#DEC_*+;     # remove all keywords beginning with DEC_
  -#TUNIT#+;    # remove all keywords TUNIT ending w. number
Note that, as a 0-or-more matching pattern, this form will succeed if the requested column or keyword is not present. In that case, the deletion expression will silently proceed as if no deletion was requested.


   [col Time, rate]              - only the Time and rate columns will
                                   appear in the filtered input file.

   [col Time, *raw]              - include the Time column and any other
                                   columns whose name ends with 'raw'.

   [col -TIME, Good == STATUS]   - deletes the TIME column and
                                   renames the status column to 'Good'

   [col PI=PHA * 1.1 + 0.2; #TUNIT#(column units) = 'counts';*]
                                 - creates new PI column from PHA values
                                   and also writes the TUNITn keyword
                                   for the new column.  The final '*'
                                   expression means preserve all the
                                   columns in the input table in the
                                   virtual output table;  without the '*'
                                   the output table would only contain
                                   the single 'PI' column.

   [col rate = rate/exposure; TUNIT#(&) = 'counts/s';*]
                                 - recomputes the rate column by dividing
                                   it by the EXPOSURE keyword value. This
                                   also modifies the value of the TUNITn
                                   keyword for this column. The use of the
                                   '&' character for the keyword comment
                                   string means preserve the existing
                                   comment string for that keyword. The
                                   final '*' preserves all the columns
                                   in the input table in the virtual
                                   output table.