Managing NICER Filter Files

Overview

All of the NICER screening criteria are based on a single file that collects key quantities in one place. This is called the "filter file" or MKF file. This document describes what is in the filter file, how it has changed over the NICER mission lifetime, and how to merge many filter files.

Read this thread if you want to: Understand NICER filter files

Last update: 2023-07-12

Introduction

Like most space-based astronomical observatories, NICER provides a way to filter and screen scientific data. This screening is based on objective criteria that the NICER team recommends, which should remove "bad" data and keep "good" data. In the case of NICER, the screening quantities are kept in a file known as the "filter file," also known as the "MKF" file.

The filter file resides in the observation directory under the auxil/ subdirectory. You will find it has a name of the form, NNNNNNNNNN/auxil/niNNNNNNNNNN.mkf where NNNNNNNNNN is the 10-digit observation ID. As retrieved from the archive, the filter file may also have a .gz suffix, which indicates that the file is gzip-compressed.

The NICER filter file contains many quantities which may be useful for screening. Over time, the contents of the file have changed. This document describes the contents and the changes.

Calibrated versus Uncalibrated Filter File

NICER maintains a notion of "uncalibrated" and "calibrated" filter files. In fact, both sets of quantities can reside in the same MKF file, with different extensions. The calibrated extension of the filter file contains more useful quantities than the uncalibrated one, including count rates that can only be derived from the calibrated event files.

If we look at an uncalibrated filter file with a tool like ftlist, it will show us something like this. Name Type Dimensions ---- ---- ---------- HDU 1 Primary Array Null Array HDU 2 PREFILTER BinTable 64 cols x 6857 rows An uncalibrated file, by itself, contains only a single extension, with about 60-70 columns. This file is created by the NICER task niprefilter. This task is run by the NICER pipeline internally and is typically not available for scientists to use. However, its operation is documented by typing "fhelp niprefilter".

Now, if we look at a calibrated filter file, it will look like this: Name Type Dimensions ---- ---- ---------- HDU 1 Primary Array Null Array HDU 2 PREFILTER BinTable 95 cols x 6857 rows HDU 3 ORIG_PREFILTER BinTable 64 cols x 6857 rows This file has two table extensions, PREFILTER (calibrated) and ORIG_PREFILTER (uncalibrated). The ORIG_PREFILTER extension is the original archive version of the table created by niprefilter as described above. The PREFILTER extension contains additional calibrated columns, as created by the NICER task niprefilter2.

niprefilter2 takes the original uncalibrated filter file quantities, and adds new quantities such as count rates in specific energy bands, based upon the X-ray event files that have already been calibrated. niprefilter2 is meant to be run by scientists, either stand-alone, or as a part of the general NICER Level 2 processing task named nicerl2, which will ensure that tasks are run in the correct order. Generally, if you are starting out with NICER analysis, you should begin with nicerl2 rather than trying to run niprefilter2 by yourself.

Time in NICER Filter Files

NICER filter files are sampled once per second. Housekeeping quantities are intended to provide the value sampled at the TIME value indicated. Counts typically will refer to the 1-second time bin that starts with the indicated TIME.

A NICER observational dataset that you download from the archive, also known as an observation segment, will typically consist of one or more observing snapshots in a single calendar UTC day. If more than one snapshot is included, the filter file table will have gaps between the snapshots.

Contents of NICER Filter Files

Here we summarize the documentation of the contents of NICER filter files for reference purposes.

Uncalibrated filter files contain columns derived from the tasks prefilter and niprefilter. For more information, please see the on-line help information for each task by following the indicated links.

Task	Calibrated?	Derived values
prefilter	No	Mission-generic quantities
niprefilter	No	NICER-specific quantities derived from housekeeping
niprefilter2	Yes	NICER-specific quantities derived from calibrated event files

Changes to Contents of the Filter File

Over time, the NICER team has modified the software to change the contents of the calibrated filter file to improve its usefulness to scientists. The following table summarizes the history of the changes.

Date	HEASoft Version	Changes
2017-07-15	6.22	First version of NICER software released (NICERDAS v1)
2018-04-24	6.24	niprefilter2 first released. First calibrated filter files
2018-10-22	6.25	ROBO_STATE column added. TIME column calibrated. Addition of LEAPINIT keyword to calibrated filter file
2020-30-26	6.27, 6.27.2	Addition of the following columns to the uncalibrated filter file: EAST_ANGLE LOCAL_TIME MAGFIELD MAGFIELD_MIN MAG_ANGLE
2021-07-16	6.29	Addition of the following columns to the uncalibrated filter file: XTI_PNT_JITTER (pointing jitter); ANG_DIST_{X,Y} (angular offset in instrument X,Y coordinates) FPM_DEADTIME (per-FPM deadtime); MPU_LOWMEM_{FIFO,SCI}_DELTA (indication of on-board data overflow); HV_ON (indicates if bias voltage is enabled) FPM_SLOW_LLD (current value of low energy threshold setting); MPU_NOISE20_COUNT (counts below 200 eV); MEDIAN_UNDERONLY_COUNT (undershoot count rate, median of array); BETA_ANGLE (orbital beta angle); TIME_SINCE_SUNSET (time in seconds since beginning of orbit night); AE8MIN,AP8MIN - AE8/AP8 modeled electron/proton fluxes
2022-10-20	6.31	Addition of the following columns("NICERV4"): MPU_XRAY_PI_0030_0070 (0.3-0.7 keV rate in each detector, used to find noise) Renaming SOLARPHI to SOLAR_PHI Removal of the following columns("NICERV4"): {MPU,TOT}_{XRAY,OVER,UNDER,ALL}_COUNT (these counts not useful) MPU_{FT,NOISE25}_{PI,PI_FAST}_{AVG,ERR} - (not used by downstream software and costly to compute) The 3C50-related columns FPM_RATIO_REJ_COUNT and MPU_NOISE20_COUNT are now always computed.
2023-07-11	6.32	Addition of the following columns("NICERV5"): {FPM,MPU}_NOISERING_COUNTS - tabulation of potential "noise ringer" events SUN_BODY_AZIMUTH - the azimuth or clock angle of sun in NICER body coordinates TOT_LOWMEM_{SCI,FIFO} - total array counts to diagnose lost events DELTA_SLOW_LLD - array-averaged low energy threshold changes Changes to the following columns: MPU_LOWMEM_{FIFO,SCI}_DELTA - are upgraded to 16-bit storage type

Note that it is possible to use the newest software to re-run the associated NICER tasks and generate a most up-to-date version of the calibrated filter file.

Dealing with Filter Files from Many Observations

It is common for requested NICER observational visits to span many segments. Any requested observational visit that exceeds one calendar day is highly likely to be split into multiple segments. In addition, a long-term monitoring campaign of a target will also consist of many visits, by definition. Therefore, scientists will naturally want to deal with filter files from many observational datasets as well.

If all filter files are of the same format, this will be fine. Scientists can merge many filter files using ftmerge without problem. However, if multiple filter files have different numbers of columns, the merging task ftmerge will refuse to combine them.

How to handle this? The NICER team recommends the following strategies.

• Use nimkfmerge. Introduced in HEASoft 6.29 (NICERDAS 8), the nimkfmerge task will readily combine filter files processed by different software versions. If nimkfmerge is set to work on filter files with mismatched columns, the rows with missing columns will be filled with NULL values.
• Run nicerl2. The nicerl2 task will re-calibrate all observations in addition to generating the most up-to-date version of the filter file. This is a more time-consuming task, but it is more fool-proof and most likely to produce correct results.
• Use ftmerge.Standard tools like ftmerge can be used to merge FITS table files. However, since nimkfmerge is more robust, including handling time ordering and updating various metadata keywords, we don't normally recommend using ftmerge any longer.

Merging Filter Files With 'nimkfmerge'

Merging filter files is straightforward with the tool 'nimkfmerge'. It's operation is similar to 'ftmerge' but handles more cases. However, please consider the tool 'niobsmerge' (available in HEASoft 6.32, July 2023), which also merge other observation data products at the same time. Internally, niobsmerge runs nimkfmerge for you.

While nimkfmerge is a useful tool, consider combining all observation products with the niobsmerge tool instead.

Still, if you insist on using nimkfmerge, here is how.

First, create a list file containing the names of filter files that you wish to merge. We can use the 'ls' command to do this. ls */auxil/ni*.mkf > filter.lis Here we have used the wildcard pattern */auxil/ni*.mkf to select filter files inside the observation directories in their standard places. You can tailor the wildcard pattern to match the observations or files that you want.

For example, if the filter files are gzip compressed, then you can add a .gz to the end of the wildcard patter. Or, if you only want to merge certain observations, say all observations beginning with 12345678, you can use a directory wildcard pattern such as 12345678?? (where the ? is a wildcard pattern that matches a single digit).

Next, we use the 'nimkfmerge' to combine the filter files: nimkfmerge infiles=@filter.lis outfile=merged.mkf clobber=YES where

• infiles is the name of the list file that we just created
• outfile is the name of the resulting merged filter file

The result is a new FITS table file, merged.mkf, which contains the combination of all input tables. Note that the input file is time-sorted by nimkfmerge.

Next Steps

You can use a time selection tool such as nimaketime on the resulting merged filter file. The result will be a GTI file that applies to the entire combined data set.

Modifications

• 2020-05-06 - initial draft
• 2021-04-16 - add navigation bar
• 2021-07-16 - add HEASoft 6.29 (NICERDAS 8) changes
• 2022-10-17 - add HEASoft 6.30 (NICERDAS 10) changes
• 2023-07-11 - add HEASoft 6.32 (NICERDAS 11) changes; tutorial on using nimkfmerge