Managing NICER Filter Files
All of the NICER screening criteria are based on a single file that collects key quantities in one place. This is called the "filter file" or MKF file. This document describes what is in the filter file, how it has changed over the NICER mission lifetime, and how to merge many filter files.
Read this thread if you want to: Understand NICER filter files
Last update: 2023-07-12
Like most space-based astronomical observatories, NICER provides a way to filter and screen scientific data. This screening is based on objective criteria that the NICER team recommends, which should remove "bad" data and keep "good" data. In the case of NICER, the screening quantities are kept in a file known as the "filter file," also known as the "MKF" file.
The filter file resides in the observation directory under the
auxil/ subdirectory. You will find it has a name of the form,
The NICER filter file contains many quantities which may be useful for screening. Over time, the contents of the file have changed. This document describes the contents and the changes.
Calibrated versus Uncalibrated Filter FileNICER maintains a notion of "uncalibrated" and "calibrated" filter files. In fact, both sets of quantities can reside in the same MKF file, with different extensions. The calibrated extension of the filter file contains more useful quantities than the uncalibrated one, including count rates that can only be derived from the calibrated event files.
If we look at an uncalibrated filter file with a tool like ftlist,
it will show us something like this.
Now, if we look at a calibrated filter file, it will look like this:
niprefilter2 takes the original uncalibrated filter file quantities, and adds new quantities such as count rates in specific energy bands, based upon the X-ray event files that have already been calibrated. niprefilter2 is meant to be run by scientists, either stand-alone, or as a part of the general NICER Level 2 processing task named nicerl2, which will ensure that tasks are run in the correct order. Generally, if you are starting out with NICER analysis, you should begin with nicerl2 rather than trying to run niprefilter2 by yourself.
Time in NICER Filter FilesNICER filter files are sampled once per second. Housekeeping quantities are intended to provide the value sampled at the TIME value indicated. Counts typically will refer to the 1-second time bin that starts with the indicated TIME.
A NICER observational dataset that you download from the archive, also known as an observation segment, will typically consist of one or more observing snapshots in a single calendar UTC day. If more than one snapshot is included, the filter file table will have gaps between the snapshots.
Contents of NICER Filter Files
Here we summarize the documentation of the contents of NICER filter files for reference purposes.
Uncalibrated filter files contain columns derived from the tasks prefilter and niprefilter. For more information, please see the on-line help information for each task by following the indicated links.
Changes to Contents of the Filter FileOver time, the NICER team has modified the software to change the contents of the calibrated filter file to improve its usefulness to scientists. The following table summarizes the history of the changes.
Note that it is possible to use the newest software to re-run the associated NICER tasks and generate a most up-to-date version of the calibrated filter file.
Dealing with Filter Files from Many ObservationsIt is common for requested NICER observational visits to span many segments. Any requested observational visit that exceeds one calendar day is highly likely to be split into multiple segments. In addition, a long-term monitoring campaign of a target will also consist of many visits, by definition. Therefore, scientists will naturally want to deal with filter files from many observational datasets as well.
If all filter files are of the same format, this will be fine. Scientists can merge many filter files using ftmerge without problem. However, if multiple filter files have different numbers of columns, the merging task ftmerge will refuse to combine them.
How to handle this? The NICER team recommends the following strategies.
Merging Filter Files With 'nimkfmerge'
Merging filter files is straightforward with the tool 'nimkfmerge'. It's operation is similar to 'ftmerge' but handles more cases. However, please consider the tool 'niobsmerge' (available in HEASoft 6.32, July 2023), which also merge other observation data products at the same time. Internally, niobsmerge runs nimkfmerge for you.
While nimkfmerge is a useful tool, consider combining all observation products with the niobsmerge tool instead.
Still, if you insist on using nimkfmerge, here is how.
First, create a list file containing the names of filter files that
you wish to merge. We can use the 'ls' command to do this.
For example, if the filter files are gzip compressed, then you can add a .gz to the end of the wildcard patter. Or, if you only want to merge certain observations, say all observations beginning with 12345678, you can use a directory wildcard pattern such as 12345678?? (where the ? is a wildcard pattern that matches a single digit).
Next, we use the 'nimkfmerge' to combine the filter files:
The result is a new FITS table file, merged.mkf, which contains the combination of all input tables. Note that the input file is time-sorted by nimkfmerge.
You can use a time selection tool such as nimaketime on the resulting merged filter file. The result will be a GTI file that applies to the entire combined data set.