Row Filtering Syntax

When entering the name of a FITS table that is to be opened by a program, an optional row filter may be specified to select a subset of the rows in the table. A virtual FITS file is created by CFITSIO on the fly (usually in memory) that contains only those rows for which the row filter expression evaluates to true. The primary array and any other extensions in the input file are also copied to the virtual file.

The row filter expression is enclosed in square brackets following the file name and extension name (e.g., 'file.fits[events][GRADE==50]' selects only those rows where the GRADE column value equals 50). When dealing with tables where each row has an associated time and/or 2D spatial position, the row filter expression can also be used to select rows based on the times in a Good Time Intervals (GTI) extension, or on spatial position as given in a SAO-style region file.

The row filtering expression can be an arbitrarily complex series of operations performed on constants, keyword values, and column data taken from the specified FITS TABLE extension. The expression must evaluate to a boolean value for each row of the table, where a value of FALSE (or zero) means that the row will be excluded.

For complex or commonly used filters, one can place the expression into a text file and import it into the row filter using the syntax '[@filename.txt]'. The expression can be arbitrarily complex and extend over multiple lines of the file. Any lines in the external text file that begin with 2 slash characters ('//') will be ignored and may be used to add comments into the file.

Keyword and column data are referenced by name. Any string of characters not surrounded by quotes (ie, a constant string) or followed by an open parentheses (ie, a function name) will be initially interpreted as a column name and its contents for the current row inserted into the expression. If no such column exists, a keyword of that name will be searched for and its value used, if found. To force the name to be interpreted as a keyword (in case there is both a column and keyword with the same name), precede the keyword name with a single pound sign, '#', as in '#NAXIS2'. Due to the generalities of FITS column and keyword names, if the column or keyword name contains a space or a character which might appear as an arithmetic term then enclose the name in '$' characters as in $MAX PHA$ or #$MAX-PHA$. Names are case insensitive.

To access a table entry in a row other than the current one, follow the column's name with a row offset within curly braces. For example, 'PHA{-3}' will evaluate to the value of column PHA, 3 rows above the row currently being processed. One cannot specify an absolute row number, only a relative offset. Rows that fall outside the table will be treated as undefined, or NULLs.

When using row filtering to open a file ``on the fly,'' it is permitted to use multiple row filtering expressions. For example, the expression

  filename.fits[#ROW > 5][X.gt.7]
would be treated as equivalent to joining the expressions with logical ``and'' like this,
  filename.fits[(#ROW > 5)&&(X.gt.7)]
Please note that if multiple row filtering expressions are used, it is not permitted to also use the \verb+[@filename.txt]+ syntax in any of the individual expressions.

Boolean operators can be used in the expression in either their Fortran or C forms. The following boolean operators are available:

    "equal"         .eq. .EQ. ==  "not equal"          .ne.  .NE.  !=
    "less than"     .lt. .LT. <   "less than/equal"    .le.  .LE.  <= =<
    "greater than"  .gt. .GT. >   "greater than/equal" .ge.  .GE.  >= =>
    "or"            .or. .OR. ||  "and"                .and. .AND. &&
    "negation"     .not. .NOT. !  "approx. equal(1e-7)"  ~
Note that the exclamation point, '!', is a special UNIX character, so if it is used on the command line rather than entered at a task prompt, it must be preceded by a backslash to force the UNIX shell to ignore it.

All of the usual mathematical operators and functions may be used in the expression. There are also many other special functions available for doing bit masking, for operating on vector columns, and for doing spatial region file and temporal good time interval filtering. See the calc_express help file for a complete description of all the functions.

Examples

    [ binary && mag <= 5.0]        - Extract all binary stars brighter
                                     than  fifth magnitude (note that
                                     the initial space is necessary to
                                     prevent it from being treated as a
                                     binning specification)

    [#row >= 125 && #row <= 175]   - Extract row numbers 125 through 175

    [IMAGE[4,5] .gt. 100]          - Extract all rows that have the
                                     (4,5) component of the IMAGE column
                                     greater than 100

    [abs(sin(theta * #deg)) < 0.5] - Extract all rows having the
                                     absolute value of the sine of theta
                                     less  than a half, where the angles
                                     are tabulated in degrees

    [SUM( SPEC > 3*BACKGRND )>=1]  - Extract all rows containing a
                                     spectrum, held in vector column
                                     SPEC, with at least one value 3
                                     times greater than the background
                                     level held in a keyword, BACKGRND

    [VCOL=={1,4,2}]                - Extract all rows whose vector column
                                     VCOL contains the 3-elements 1, 4, and
                                     2.

    [@rowFilter.txt]               - Extract rows using the expression
                                     contained within the text file
                                     rowFilter.txt

    [gtifilter()]                  - Search the current file for a GTI
				     extension,  filter  the TIME
				     column in the current table, using
				     START/STOP times taken from
				     columns in the GTI  extension

    [regfilter("pow.reg")]         - Extract rows which have a coordinate
                                     (as given in the X and Y columns) 
                                     within the spatial region specified
                                     in the pow.reg region file. 

    [regfilter("pow.reg", Xs, Ys)] - Same as above, except that the
                                     Xs and Ys columns will be used
                                     to determine the coordinate of each
                                     row in the table.

SEE ALSO

filenames, colfilter, imfilter, pixfilter. binfilter. calc_express.