Screening Data with FPM Selections

Overview

NICER's analysis software introduced a new concept in HEASoft 6.29 (NICERDAS 8) called "FPM Selection" information. This information allows the analyst to select and screen their NICER data, while still maintaining the accounting information needed to generate response products.

Read this thread if you want to: Screen Data by Time or Detector Selection

Last update: 2023-02-15

Introduction

In the previous analysis thread, What is FPM Selection Information?, you were introduce to FPM Selection information. Please see that article if you wish to know what FPM Selection is, how it is used at a high level, and where it is stored.

In this article we will discuss how to actually screen data and maintain FPM Selection information up to date.

Why Would You Want To Perform FPM Screening?

First, let's establish why an analyst would want to screen data by FPM.

This is a topic unto itself, but the typical use case will be to exclude certain detectors. Typically these detectors will be considered to be "noisy" or undesireable in some way, and the analyst wishes to prevent data from those detectors from overwhelming the data from "good" detectors. Deciding which detectors are "good" or not is an art that we will not go into in this thread, but it is an art.

Thus, in its typical application, a user will want to say, "remove detector XX from my event file," where XX is one (or more) out of the 52 operable detectors. On the other hand, in rarer cases, the user may want to consider only certain detectors and exclude all others. We call this selecting and deselecting detectors.

Note that a deselecting or selecting detectors is different than which detectors are on/enabled or off/disabled. Enablement/disablement represents the state of the hardware and whether it is on and producing science events. Selecting or deselecting detectors is a human judgement that certain detectors are of interest or not of interest. Of course, by nature, "selecting" a detector which is off is not useful because no events can come from an off detector. But deselecting a noisy detector removes its events from the event file

In this example we will consider removing detector 34 from the event file. As discussed below, there are many more sophisticated ways to select/deselect an event file, but this simple example will carry us forward. Figure 1 shows an example flow using nifpmsel.
Figure 1. Example nifpmsel.

One can see that an original event file, will be the input, and after running nifpmsel, an new event file will be produced after excluding detector 34.

Can't I just use ftselect or fselect?

You may be asking, can't I just use my existing workflow that utilizes tools such as fselect or ftselect to remove certain events from my event file?

The short answer is, YES, you can continue to use your existing workflow. You are not forced to use nifpmsel.

However, if you do this, what you lose is the ability to automatically account for which detectors are included/excluded from an event file, and thus when you make response files, you will be responsible to manually set a detector list.

Prerequisites

We will assume the following prerequisites.

• The user has an event file (example myevent.evt). This file should have FPM Selection information included, which means that it must have been created by nicerl2 from HEASoft 6.29 (NICERDAS 8) or later.
• The user knows which detector(s) to exclude (excluding 14 and 34 in this example).

To generate a new event file with detectors 14 and 34 deselected, use the following command. nifpmsel myevent.evt myevent_sel.evt detlist=launch,-14,-34 where

• myevent.evt is your event file
• myevent_sel.evt is the resulting output
• detlist gives which detectors to select or deselect.

The 'detlist' parameter specifies which detectors should be selected or deselected. It is a comma-separated list that should be read from left to right, in that order. Minus signs mean to deselect (exclude) a requested detector ID number. In this case, "detlist=launch,-14,-34" means, starting with the launch configuration of 52 operable detectors, deselect detectors 14 and 34.

Please note that if a detector is off, there is no way to select data from it. Likewise, once events from a detector are removed nifpmsel, there is no way to recover them, except by returning to the original event file. However, performing such non-sensical operations will still produce a valid event file with valid FPM Selection information that can be used by the response generators.

In this example, the output file is myevent_sel.evt, which is a new file, and the original file myevent.evt is kept intact. It is possible to set the output file to "INFILE", in which case the input file is screened and overwritten in place. Although this may be convenient, it may be dangerous if you make a mistake. Thus, for most usage, we recommend that users make a new output file.

Deselection and Filtering Details

The deselection of FPMs refers to the updating of FPM selection information by which certain detectors are newly designated as OFF. These selections are accounted for in downstream applications such as the NICER RMF and ARF generators. FPM-based filtering refers to the removal of events from a specified detector or detectors. It is possible to deselect an FPM without removing the events in that FPM, and vice-versa, though generally users will want to keep these synchronized. The behaviors explained below pertain to NICERDAS version 8, and are expected to change due to software fixes in subsequent updates.

• The NICERDAS 8 version of nicerl2 task deselects FPMs if requested via the nicerl2 "detlist" parameter, but does no FPM-based filtering of any event file.
• The nifpmsel task deselects FPMs - if they have not already been deselected - based on whether they are excluded or not included according to the "detlist" parameter. So, nifpmsel myevent.evt myevent_sel.evt detlist=launch,-14,-34 deselects detectors 14 and 34, and nifpmsel myevent.evt myevent_sel.evt detlist=00,01,13,67 deselects all detectors except 00,01,13, and 67.
• The nifpmsel task never reverses any previous deselection. So, detectors 14 and 34 remain deselected in the output from the command nicerl2 indir=mydir detlist=launch,-14,-34 followed by a run of nifpmsel, regardless of the setting of the nifpmsel "detlist" parameter for the second run (i.e., even if detectors 14 and 34 are not excluded).
• The NICERDAS 8 version of nifpmsel task only filters events from newly deselected FPMs. So, events from detectors 14 and 34 are filtered in the output from the pair of commands nicerl2 indir=mydir detlist=launch nifpmsel myevent.evt myevent_sel.evt detlist=launch,-14,-34
• However, the NICERDAS 8 version of nifpmsel task never filters events in previously deselected FPMs. So, no (additional) events are filtered in the output from the pair of commands nicerl2 indir=mydir detlist=launch,-14,-34 nifpmsel myevent.evt myevent_sel.evt detlist=launch,-14,-34 and only events from detector 34 are filtered in the output from the pair of commands nicerl2 indir=mydir detlist=launch,-14 nifpmsel myevent.evt myevent_sel.evt detlist=launch,-14,-34
• We recommend, for NICERDAS version 8 and earlier, that users implement FPM-based filtering of an event file by first running nicerl2 with "detlist=launch", and then running nifpmsel with "detlist" set as needed. For example, nicerl2 indir=mydir detlist=launch clobber=yes nifpmsel myevent.evt mevent_sel.evt detlist=launch,-7,-10,-13,-14,-32,-34 clobber=yes

What's Next

At this point, if you have successfully screened your data, you can extract a spectrum or light curve using 'xselect' or your favorite extraction tool.

Advanced Usage

In the following sections, we will discuss advanced usage of nifpmsel. The topics covered will be more in-depth discussion of how to use the 'detlist' parameter to select or deselect detectors, as well as to perform actions over specific time ranges only.

How to Screen by Detector and/or Time with "detlist"

The key benefit of nifpmsel is the ability to screen by detector, and by time, using a single detector screening expression, and track the exposure accurately. To do this, nifpmsel allows a powerful set of screenings to by applied from the command line using the 'detlist' parameter. This section describes how to use the detlist parameter.

Essentially, 'detlist' is a comma-separated list of detectors to include or exclude from processing. The list is read from left to right. Hence, a detlist can look like this,

   detlist=A,B,C,D,E,...

The individual list elements can be simple or complex. For basic processing, detlist can be simply a list of detectors to include or exclude. Because advanced usage of detlist selections may use special characters, it is recommended to use your operating system's quoting characters. For example, on the command line,

   detlist="A,B,C,D,..."

although for simple usage this may not be needed. In the examples below, the quotes have been removed for simplicity.

Including Detectors. Select individual detectors by listing the detector number as a single item. Example:

      detlist=00,01,13,67

would select detectors 00, 01, 13 and 67, and not any others.

Including Detectors Shorthand Notation. Typically the user may want to select "all" detectors. nifpmsel allows some shorthand notations: "launch" to indicate all detectors operating at launch (which is all detectors, excluding detectors 11, 20, 22, and 60), and "all" to select all detectors. Note that since nifpmsel can never add detectors that were never selected to begin with, "all" and "launch" are effectively the same thing. Hence, this example,

    detlist=launch

would select all pre-launch operational detectors.

Including Detectors by Range or MPU. The user may select detectors in a range by listing the range as a dash separator. The user may also select all detectors in a given MPU with "MPUn" where n may be between 0 and 6. In this example,

   detlist=MPU0,10,12-17

would include all detectors in MPU0 (00-07), detector 10, and detectors 12-17.

Excluding Detectors. The user may also exclude individual detectors by listing them preceded by a minus sign. This may be desireable if certain detectors are known to be "bad" for the entire observation duration. Thus, in this example,

   detlist=launch,-14,-34

The selection begins with the set of all detectors at launch, and deselects detectors 14 and 34 from that set. Remember that the list is read from left to right, so the last element in the list pertaining to a certain detector takes precedence.

Including Detectors by GTI. All of the ways to select individual detectors and MPUs also allow to filter by time using a GTI file. The notation is detetector(gtifile:filename) and MPUnumber(gtifile:filename). The filename should be a valid NICER GTI file and extension number. Examples,

  detlist=launch,14(gtifile:det_14.gti[1]),34(gtifile:det_34.gti[1])
  detlist=launch,MPU3(gtifile:mpu3.gti[1])

In the first example, all detectors are selected, and detector 14 is filtered using the first table extension of det_14.gti, and detector 34 is filtered using det_34.gti. In the second example, all detectors in MPU3 are filtered using mpu3.gti. Note that the "[1]" is required because CFITSIO does not automatically open GTI extensions.

Note that it is possible to include multiple selections for the same detector by listing them separately in the list. The resulting filtering is the intersection of all specified GTI files. Examples,

  detlist=launch,14(gtifile:det_14a.gti[1]),14(gtifile:det_14b.gti[1])
  detlist=launch,MPU3(gtifile:mpu3.gti[1]),34(gtifile:det_34.gti[1])

In the first example, detector 14 is filtered by the intersection of GTI files det_14a.gti and det_14b.gti. In the second example, all of the detectors in MPU3 are selected by mpu3.gti, and detector 34 is selected by the intersection of both mpu3.gti and det_34.gti.

Including Detectors by Explicit Time Range. As a convenience, anywhere where a GTI file is can be given, it is also possible to list NICER MET time ranges explicitly on the command line. The form is detector_or_MPU(gti:start-). Internally, nifpmsel converts this to a GTI file. The time values should be the true, calibrated TIME range values (i.e. after correcting for TIMEZERO). Multiple time ranges can be separated by the pipe symbol. Examples:

  detlist=launch,14(gti:123768005.0-123768555.0|123770217.0-12371217.0)
  detlist=launch,MPU3(gti:123768005.0-123768555.0|123770217.0-12371217.0)

In the first example, detector 14 is filtered to include only data between the two given time ranges. In the second example, all detectors in MPU3 are filtered between the given time ranges.

Excluding Detectors by Explicit Time Range. In addition, it is possible to exclude detectors or whole MPUs by a given time range on the command line. The form is detector_or_MPU(bti:start-). Internally, nifpmsel converts this to a GTI file that will exclude the given time ranges. The time values should be the true, calibrated TIME range values (i.e. after correcting for TIMEZERO). Multiple time ranges can be separated by the pipe symbol. Examples:

  detlist=launch,14(bti:123768005.0-123768555.0|123770217.0-12371217.0)
  detlist=launch,MPU3(bti:123768005.0-123768555.0|123770217.0-12371217.0)

In the first example, detector 14 is filtered to excclude data between the two given time ranges, and include any other time ranges when the detector is enabled. In the second example, all detectors in MPU3 are filtered to exclude the given time ranges.

How Multiple Time Ranges Are Combined. As noted above, multiple time selections can be given for a single detector. The expression will be read from left to right, and each detectors' time selections will be listed in that order. In addition, any filtering expressions applied from previous invocations of nifpmsel are also added to the list. Finally the overall event-level GTI is added to the list. When the parsing of the 'detlist' expression is complete, the time filters are combined using a GTI merge using "AND" logic (i.e. ftmgtime with operation=AND).

Thus, the result of all filtering is the most restrictive combination of both the user-specified filters as well as any filters that have been already applied to the input file.

Specifying Non-Sensical Selections. Since the detlist expression can be rather complex, it is possible to specify potentially non-sensical items. For example, this expression,

  detlist=launch,-14,14

may seem non-sensical because it both deselects and selects detector 14. How will nifpmsel interpret this? As noted above, nifpmsel reads the expression from left to right, so while the "-14" will be read to deselect detector 14, the next expression will cause detector "14" to be selected again. The last state is the one that is used by nifpmsel. In this next example,

  detlist=launch,-14,14(gti:123768005.0-123768555.0)

the first item deselects detector "14" but, the next item causes it to be selected again, with the given time range. In this final example,

  detlist=launch,14(gti:123768005.0-123768555.0),-14

The first item selects detector 14 in the given time range, but the second item deselects detector 14 entirely.

nifpmsel does error checking, but cannot always ascertain if a certain behavior is undesireable. Since these behaviors could be confusing or "abused", analysts should keep their selection expressions as simple as possible.

Modifications

• 2020-07-16 - initial draft
• 2022-01-16 - added "details" section