10.10 Column and Keyword Filtering Specification

The optional column/keyword filtering specifier is used to modify the column structure and/or the header keywords in the HDU that was selected with the previous HDU location specifier. This filtering specifier must be enclosed in square brackets and can be distinguished from a general row filter specifier (described below) by the fact that it begins with the string 'col ' and is not immediately followed by an equals sign. The original file is not changed by this filtering operation, and instead the modifications are made on a copy of the input FITS file (usually in memory), which also contains a copy of all the other HDUs in the file. (If a `#' character is appended to the name or number of the table HDU then only the primary array, and none of the other HDUs in the input file will be copied into memory). This temporary file is passed to the application program and will persist only until the file is closed or until the program exits, unless the outfile specifier (see above) is also supplied.

The column/keyword filter can be used to perform the following operations. More than one operation may be specified by separating them with commas or semi-colons.

• Copy only a specified list of columns columns to the filtered input file. The list of column name should be separated by commas or semi-colons. Wild card characters may be used in the column names to match multiple columns. If the expression contains both a list of columns to be included and columns to be deleted, then all the columns in the original table except the explicitly deleted columns will appear in the filtered table (i.e., there is no need to explicitly list the columns to be included if any columns are being deleted).

• Delete a column or keyword by listing the name preceded by a minus sign or an exclamation mark (!), e.g., '-TIME' will delete the TIME column if it exists, otherwise the TIME keyword. An error is returned if neither a column nor keyword with this name exists. Note that the exclamation point, '!', is a special UNIX character, so if it is used on the command line rather than entered at a task prompt, it must be preceded by a backslash to force the UNIX shell to ignore it.

• Rename an existing column or keyword with the syntax 'NewName == OldName'. An error is returned if neither a column nor keyword with this name exists.

• Append a new column or keyword to the table. To create a column, give the new name, optionally followed by the data type in parentheses, followed by a single equals sign and an expression to be used to compute the value (e.g., 'newcol(1J) = 0' will create a new 32-bit integer column called 'newcol' filled with zeros). The data type is specified using the same syntax that is allowed for the value of the FITS TFORMn keyword (e.g., 'I', 'J', 'E', 'D', etc. for binary tables, and 'I8', F12.3', 'E20.12', etc. for ASCII tables). If the data type is not specified then an appropriate data type will be chosen depending on the form of the expression (may be a character string, logical, bit, long integer, or double column). An appropriate vector count (in the case of binary tables) will also be added if not explicitly specified.

  When creating a new keyword, the keyword name must be preceded by a pound sign '#', and the expression must evaluate to a scalar (i.e., cannot have a column name in the expression). The comment string for the keyword may be specified in parentheses immediately following the keyword name (instead of supplying a data type as in the case of creating a new column). If the keyword name ends with a pound sign '#', then cfitsio will substitute the number of the most recently referenced column for the # character . This is especially useful when writing a column-related keyword like TUNITn for a newly created column, as shown in the following examples.

  COMMENT and HISTORY keywords may also be created with the following syntax:

     #COMMENT = 'This is a comment keyword'
     #HISTORY = 'This is a history keyword'
  

  Note that the equal sign and the quote characters will be removed, so that the resulting header keywords in these cases will look like this:

     COMMENT This is a comment keyword
     HISTORY This is a history keyword
  

  These two special keywords are always appended to the end of the header and will not affect any previously existing COMMENT or HISTORY keywords.

  It is possible to delete an existing keyword using a preceding '-'. Either of these examples will delete the keyword named VEL.

    -VEL;
    -#VEL;
  

• Recompute (overwrite) the values in an existing column or keyword by giving the name followed by an equals sign and an arithmetic expression.

The expression that is used when appending or recomputing columns or keywords can be arbitrarily complex and may be a function of other header keyword values and other columns (in the same row). The full syntax and available functions for the expression are described below in the row filter specification section.

If the expression contains both a list of columns to be included and columns to be deleted, then all the columns in the original table except the explicitly deleted columns will appear in the filtered table. If no columns to be deleted are specified, then only the columns that are explicitly listed will be included in the filtered output table. To include all the columns, add the '*' wildcard specifier at the end of the list, as shown in the examples.

For complex or commonly used operations, one can place the operations into an external text file and import it into the column filter using the syntax '[col @filename.txt]'. The operations can extend over multiple lines of the file, but multiple operations must still be separated by commas or semi-colons. Any lines in the external text file that begin with 2 slash characters ('//') will be ignored and may be used to add comments into the file.

When using column filtering to open a file “on the fly,” it is permitted to use multiple column filtering expressions. For example, the syntax

  filename.fits[col *][col -Y][col Z=X+1]

would be treated as equivalent to joining the expressions with semicolons, or

  filename.fits[col *; -Y;col Z=X+1]

Please note that if multiple column filtering expressions are used, it is not permitted to also use the [col @filename.txt] syntax in any of the individual expressions.

It is possible to use wildcard syntax to delete either keywords or columns that match a pattern. Recall that to delete either a keyword or a column, precede its name with a '-' character.

Wildcard patterns are: '*', which matches any string of characters; '?', which matches any single character; and '#' which matches any numerical string. For example these statements:

  -VEL*;       # remove single column (or keyword) beginning with VEL
  -VEL_?;      # remove single column (or keyword) VEL_? where ? is any character
  -#DEC_*;     # remove single keyword beginning with DEC_
  -#TUNIT#;    # remove single keyword TUNIT ending w. number

will remove the columns or keywords as noted. Be aware that if a '#' is not present, the CFITSIO engine will check for columns with the given name first, followed by keywords.

The above expressions will only delete the first item which matches the pattern. If following columns or keywords in the same CHDU match the pattern, they will not be deleted. To delete zero or more keywords that match the pattern, add a trailing '+'.

  -VEL*+;       # remove all columns (or keywords) beginning with VEL
  -VEL_?+;      # remove all columns (or keyword) VEL_? where ? is any character
  -#DEC_*+;     # remove all keywords beginning with DEC_
  -#TUNIT#+;    # remove all keywords TUNIT ending w. number

Note that, as a 0-or-more matching pattern, this form will succeed if the requested column or keyword is not present. In that case, the deletion expression will silently proceed as if no deletion was requested.

Examples:

   [col Time, rate]              - only the Time and rate columns will
                                   appear in the filtered input file.

   [col Time, *raw]              - include the Time column and any other
                                   columns whose name ends with 'raw'.

   [col -TIME, Good == STATUS]   - deletes the TIME column and
                                   renames the status column to 'Good'

   [col PI=PHA * 1.1 + 0.2; #TUNIT#(column units) = 'counts';*]
                                 - creates new PI column from PHA values
                                   and also writes the TUNITn keyword
                                   for the new column.  The final '*'
                                   expression means preserve all the
                                   columns in the input table in the
                                   virtual output table;  without the '*'
                                   the output table would only contain
                                   the single 'PI' column.

   [col rate = rate/exposure; TUNIT#(&) = 'counts/s';*]
                                 - recomputes the rate column by dividing
                                   it by the EXPOSURE keyword value. This
                                   also modifies the value of the TUNITn
                                   keyword for this column. The use of the
                                   '&' character for the keyword comment
                                   string means preserve the existing
                                   comment string for that keyword. The
                                   final '*' preserves all the columns
                                   in the input table in the virtual
                                   output table.