NAME

nifpmsel - screen NICER event data by FPM (detector)

USAGE

nifpmsel infile outfile detlist [mkfile=niNNNNNNNNNN.mkf]

DESCRIPTION

The nifpmsel task screens NICER event data by Focal Plane Module (FPM) detector. NICER is composed of 56 independently operating FPMs, which can be enabled or disabled for a given observation. In addition, the analyst may wish to deselect certain detectors, or range of detectors, for all or some of the observation time. This tool allows this type of screening.

A key benefit of this tool is that it keeps track of the exposure of each FPM. This information can be used by the nicerarf or nivignette tool, which aids in the computation of accurate response products. Correct knowledge of which detectors are included or not, and when, is required to compute an accurate response.

The input to the task is a NICER event file, as produced by nicerl2, or a previous run of nifpmsel. The structure of a NICER event file processed in this way is described further below. Please note that if this is the first time to run nifpmsel, it may be required to supply the NICER filter file (.mkf file) which covers the same time range as the event file.

It is possible to iteratively screen an event file by running nifpmsel multiple times. While it is always possible to remove data (i.e. de-select FPMs for all or some time), nifpmsel cannot add data back. The analyst would need to return to the original fresh nicerl2 file to recover all data from all times.

The output of the nifpmsel task is a new event file, with any requested screening applied. It is possible to operate the task in-place, where the input file is modified directly, by setting outfile=INFILE. The nifpmtask updates any needed GTI extensions that track per-FPM exposure.

When is a Filter File Required for Input?

nifpmsel requires the NICER filter file as input, sometimes but not every time. The first time you run nifpmsel, you need to include the filter file name using the mkfile parameter. For example, mkfile=1234567890/auxil/ni1234567890.mkf. Any subsequent runs, after the event file has been augmented by nifpmsel, do not require the mkfile parameter. However, there is no harm to specifying the mkfile parameter even if it is not needed; nifpmsel will simply ignore / not open the file.

How to Screen by Detector and/or Time with "detlist"

The key benefit of nifpmsel is the ability to screen by detector, and by time, using a single detector screening expression, and track the exposure accurately. To do this, nifpmsel allows a powerful set of screenings to by applied from the command line using the 'detlist' parameter. This section describes how to use the detlist parameter.

Essentially, 'detlist' is a comma-separated list of detectors to include or exclude from processing. The list is read from left to right. Hence, a detlist can look like this,


   detlist=A,B,C,D,E,...

The individual list elements can be simple or complex. For basic processing, detlist can be simply a list of detectors to include or exclude. Because advanced usage of detlist selections may use special characters, it is recommended to use your operating system's quoting characters. For example, on the command line,
   detlist="A,B,C,D,..."
although for simple usage this may not be needed. In the examples below, the quotes have been removed for simplicity.

Including Detectors. Select individual detectors by listing the detector number as a single item. Example:

      detlist=00,01,13,67
would select detectors 00, 01, 13 and 67, and not any others.

Including Detectors: Shorthand Notation. Typically the user may want to select "all" detectors. nifpmsel allows some shorthand notations: "launch" to indicate all detectors operating at launch (which is all detectors, excluding detectors 11, 20, 22, and 60), and "all" to select all detectors. Note that since nifpmsel can never add detectors that were never selected to begin with, "all" and "launch" are effectively the same thing. Hence, this example,

    detlist=launch
would select all pre-launch operational detectors.

Including Detectors by Range or MPU. The user may select detectors in a range by listing the range as a dash separator. The user may also select all detectors in a given MPU with "MPUn" where n may be between 0 and 6. In this example,

   detlist=MPU0,10,12-17
would include all detectors in MPU0 (00-07), detector 10, and detectors 12-17.

Excluding Detectors. The user may also exclude individual detectors by listing them preceded by a minus sign. This may be desireable if certain detectors are known to be "bad" for the entire observation duration. Thus, in this example,

   detlist=launch,-14,-34
The selection begins with the set of all detectors at launch, and deselects detectors 14 and 34 from that set. Remember that the list is read from left to right, so the last element in the list pertaining to a certain detector takes precedence.

Including Detectors by GTI. All of the ways to select individual detectors and MPUs also allow to filter by time using a GTI file. The notation is detetector(gtifile:filename) and MPUnumber(gtifile:filename). The filename should be a valid NICER GTI file and extension number. Examples,

  detlist=launch,14(gtifile:det_14.gti[1]),34(gtifile:det_34.gti[1])
  detlist=launch,MPU3(gtifile:mpu3.gti[1])
In the first example, all detectors are selected, and detector 14 is filtered using the first table extension of det_14.gti, and detector 34 is filtered using det_34.gti. In the second example, all detectors in MPU3 are filtered using mpu3.gti. Note that the "[1]" is required because CFITSIO does not automatically open GTI extensions.

Note that it is possible to include multiple selections for the same detector by listing them separately in the list. The resulting filtering is the intersection of all specified GTI files. Examples,

  detlist=launch,14(gtifile:det_14a.gti[1]),14(gtifile:det_14b.gti[1])
  detlist=launch,MPU3(gtifile:mpu3.gti[1]),34(gtifile:det_34.gti[1])
In the first example, detector 14 is filtered by the intersection of GTI files det_14a.gti and det_14b.gti. In the second example, all of the detectors in MPU3 are selected by mpu3.gti, and detector 34 is selected by the intersection of both mpu3.gti and det_34.gti.

Including Detectors by Explicit Time Range. As a convenience, anywhere where a GTI file is can be given, it is also possible to list NICER MET time ranges explicitly on the command line. The form is detector_or_MPU(gti:start-). Internally, nifpmsel converts this to a GTI file. The time values should be the true, calibrated TIME range values (i.e. after correcting for TIMEZERO). Multiple time ranges can be separated by the pipe symbol. Examples:

  detlist=launch,14(gti:123768005.0-123768555.0|123770217.0-12371217.0)
  detlist=launch,MPU3(gti:123768005.0-123768555.0|123770217.0-12371217.0)
In the first example, detector 14 is filtered to include only data between the two given time ranges. In the second example, all detectors in MPU3 are filtered between the given time ranges.

Excluding Detectors by Explicit Time Range. In addition, it is possible to exclude detectors or whole MPUs by a given time range on the command line. The form is detector_or_MPU(bti:start-). Internally, nifpmsel converts this to a GTI file that will exclude the given time ranges. The time values should be the true, calibrated TIME range values (i.e. after correcting for TIMEZERO). Multiple time ranges can be separated by the pipe symbol. Examples:

  detlist=launch,14(bti:123768005.0-123768555.0|123770217.0-12371217.0)
  detlist=launch,MPU3(bti:123768005.0-123768555.0|123770217.0-12371217.0)
In the first example, detector 14 is filtered to excclude data between the two given time ranges, and include any other time ranges when the detector is enabled. In the second example, all detectors in MPU3 are filtered to exclude the given time ranges.

Reading detlist specifications from a file. One can also read specifications from a file using the notation detlist=@filename.txt. The file should be a text file with a single line of detlist specifications. The @filename notation can combined with other detlist specifications as well as multiple @filename entries within the comma-separated list. Note that this does restrict filenames from containing a comma.

How Multiple Time Ranges Are Combined. As noted above, multiple time selections can be given for a single detector. The expression will be read from left to right, and each detectors' time selections will be listed in that order. In addition, any filtering expressions applied from previous invocations of nifpmsel are also added to the list. Finally the overall event-level GTI is added to the list. When the parsing of the 'detlist' expression is complete, the time filters are combined using a GTI merge using "AND" logic (i.e. ftmgtime with operation=AND).

Thus, the result of all filtering is the most restrictive combination of both the user-specified filters as well as any filters that have been already applied to the input file.

Specifying Non-Sensical Selections. Since the detlist expression can be rather complex, it is possible to specify potentially non-sensical items. For example, this expression,

  detlist=launch,-14,14
may seem non-sensical because it both deselects and selects detector 14. How will nifpmsel interpret this? As noted above, nifpmsel reads the expression from left to right, so while the "-14" will be read to deselect detector 14, the next expression will cause detector "14" to be selected again. The last state is the one that is used by nifpmsel. In this next example,
  detlist=launch,-14,14(gti:123768005.0-123768555.0)
the first item deselects detector "14" but, the next item causes it to be selected again, with the given time range. In this final example,
  detlist=launch,14(gti:123768005.0-123768555.0),-14
The first item selects detector 14 in the given time range, but the second item deselects detector 14 entirely.

nifpmsel does error checking, but cannot always ascertain if a certain behavior is undesireable. Since these behaviors could be confusing or "abused", analysts should keep their selection expressions as simple as possible.

How a Selection Information is Stored

nifpmsel stores detector selection and deselection information in the output file. This information is stored in several output extensions. This section describes the output file format briefly.

The output of the nicerl2 pipeline creates an event file with some, but not all required extensions. The extensions in the "standard" file are:

After nifpmsel completes, the output file contains additional extensions which give additional information about which selections have been applied.

The FPM_SEL extension is more or less a master "directory" which collects a summary all filtering information into a single extension. It is a binary table with the following columns:

The FPM_SEL extension is used by the NICER ARF calculation (nicerarf and/or nivignette). In addition, the FPM_SEL extension contains an array of 56 keywords named DETSELnn, where nn is the detector ID number (00-67). The allowed values of these keywords is:

Memory and Disk Performance

This section describes how nifpmsel operates for large files. Generally speaking, nifpmsel does not require to read the entire file into memory. For the largest event files (~10 GB), perhaps 2 GB of memory is required.

nifpmsel makes working copies of data in the output directory (which are typically cleaned up at the end of the task). The task typically will require at least enough storage to keep one working copy of the entire event file.

PARAMETERS

infile [filename]
Input event file name. Any gzip-compressed event files should be unzipped before running the task.
outfile=INFILE [string]
The name of the output event file after selections have been applied. The special string INFILE means to overwrite the input file (note that for this to work, clobber=YES must also be set). The output file name should not be set to a file with .gz suffix.
detlist="launch" [string]
Detector selection expression, as described in detail above.
(mkfile="NONE") [string]
Name of NICER filter (.mkf) file, or NONE if not needed.
(gti="GTI") [string]
Name of whole-file GTI extension in the input file, which is "GTI" for NICER event files.
(filter=YES) [boolean]
If set, then apply any selections/deselections to the event file. If not set, then only update the accouting extensions. Setting to NO is advanced usage, not recommended for typical applications.
(cleanup="YES") [boolean]
If yes, then clean up temporary files. If no, temporary files remain. This is typically for debugging.
(clobber = NO) [boolean]
If the output file already exists, then setting "clobber = yes" will cause it to be overwritten.
(chatter = 2) [integer, 0 - 5]
Controls the amount of informative text written to standard output. Setting chatter = 4 or higher will produce detailed diagnostic output; chatter = 1 prints out a basic diagnostic message. The default is to produce a brief summary on output.
(history = YES) [boolean]
If history = YES, then a set of HISTORY keywords will be written to the header of the specified HDU in the output file to record the value of all the nifpmsel task parameters that were used to produce the output file.

EXAMPLES

Run nifpmsel for myevents.evt for the first time. No additional selections are performed (detlist="launch").

  nifpmsel myevents.evt INFILE launch mkfile=1234567890/auxil/ni1234567890.mkf
Note that the mkfile points to the observation filter file. The output overwrites the input since outfile=INFILE. Since no special selections are applied, this operation basically updates the accounting information stored in the FPM_SEL and GTI_* extensions, as described above.

Run nifpmsel for myevents, and exclude detectors 14 and 34,

  nifpmsel myevents.evt INFILE "launch,-14,-34"
Note that the mkfile parameter has been left out this time, assuming you have already run nifpmsel once, but you can add it if necessary (and it doesn't hurt). Operating system quotation marks have been used to bracket the detlist parameter.

Please see above for more examples of detector selection expressions.

SEE ALSO

nicerarf

nivignette

nicerl2

nicerrmf

LAST MODIFIED

Mar 2021