HEASARC Staff Scientist Position - Applications are now being accepted for a Staff Scientist with significant experience and interest in the technical aspects of astrophysics research, to work in the High Energy Astrophysics Science Archive Research Center (HEASARC) at NASA Goddard Space Flight Center (GSFC) in Greenbelt, MD. Refer to the AAS Job register for full details.

Search:

[Advanced Search]



Managing NICER Filter Files

Overview

All of the NICER screening criteria are based on a single file that collects key quantities in one place. This is called the "filter file" or MKF file. This document describes what is in the filter file, how it has changed over the NICER mission lifetime, and how to merge many filter files.

Read this thread if you want to: Understand NICER filter files

Last update: 2023-07-12

Introduction

Like most space-based astronomical observatories, NICER provides a way to filter and screen scientific data. This screening is based on objective criteria that the NICER team recommends, which should remove "bad" data and keep "good" data. In the case of NICER, the screening quantities are kept in a file known as the "filter file," also known as the "MKF" file.

The filter file resides in the observation directory under the auxil/ subdirectory. You will find it has a name of the form, NNNNNNNNNN/auxil/niNNNNNNNNNN.mkf where NNNNNNNNNN is the 10-digit observation ID. As retrieved from the archive, the filter file may also have a .gz suffix, which indicates that the file is gzip-compressed.

The NICER filter file contains many quantities which may be useful for screening. Over time, the contents of the file have changed. This document describes the contents and the changes.

Calibrated versus Uncalibrated Filter File

NICER maintains a notion of "uncalibrated" and "calibrated" filter files. In fact, both sets of quantities can reside in the same MKF file, with different extensions. The calibrated extension of the filter file contains more useful quantities than the uncalibrated one, including count rates that can only be derived from the calibrated event files.

If we look at an uncalibrated filter file with a tool like ftlist, it will show us something like this. Name Type Dimensions ---- ---- ---------- HDU 1 Primary Array Null Array HDU 2 PREFILTER BinTable 64 cols x 6857 rows An uncalibrated file, by itself, contains only a single extension, with about 60-70 columns. This file is created by the NICER task niprefilter. This task is run by the NICER pipeline internally and is typically not available for scientists to use. However, its operation is documented by typing "fhelp niprefilter".

Now, if we look at a calibrated filter file, it will look like this: Name Type Dimensions ---- ---- ---------- HDU 1 Primary Array Null Array HDU 2 PREFILTER BinTable 95 cols x 6857 rows HDU 3 ORIG_PREFILTER BinTable 64 cols x 6857 rows This file has two table extensions, PREFILTER (calibrated) and ORIG_PREFILTER (uncalibrated). The ORIG_PREFILTER extension is the original archive version of the table created by niprefilter as described above. The PREFILTER extension contains additional calibrated columns, as created by the NICER task niprefilter2.

niprefilter2 takes the original uncalibrated filter file quantities, and adds new quantities such as count rates in specific energy bands, based upon the X-ray event files that have already been calibrated. niprefilter2 is meant to be run by scientists, either stand-alone, or as a part of the general NICER Level 2 processing task named nicerl2, which will ensure that tasks are run in the correct order. Generally, if you are starting out with NICER analysis, you should begin with nicerl2 rather than trying to run niprefilter2 by yourself.

Time in NICER Filter Files

NICER filter files are sampled once per second. Housekeeping quantities are intended to provide the value sampled at the TIME value indicated. Counts typically will refer to the 1-second time bin that starts with the indicated TIME.

A NICER observational dataset that you download from the archive, also known as an observation segment, will typically consist of one or more observing snapshots in a single calendar UTC day. If more than one snapshot is included, the filter file table will have gaps between the snapshots.

Contents of NICER Filter Files

Here we summarize the documentation of the contents of NICER filter files for reference purposes.

Uncalibrated filter files contain columns derived from the tasks prefilter and niprefilter. For more information, please see the on-line help information for each task by following the indicated links.

TaskCalibrated?Derived values
prefilter NoMission-generic quantities
niprefilter NoNICER-specific quantities derived from housekeeping
niprefilter2 YesNICER-specific quantities derived from calibrated event files

Changes to Contents of the Filter File

Over time, the NICER team has modified the software to change the contents of the calibrated filter file to improve its usefulness to scientists. The following table summarizes the history of the changes.
DateHEASoft VersionChanges
2017-07-156.22 First version of NICER software released (NICERDAS v1)
2018-04-246.24 niprefilter2 first released. First calibrated filter files
2018-10-226.25 ROBO_STATE column added.
TIME column calibrated.
Addition of LEAPINIT keyword to calibrated filter file
2020-30-266.27, 6.27.2 Addition of the following columns to the uncalibrated filter file:
EAST_ANGLE
LOCAL_TIME
MAGFIELD
MAGFIELD_MIN
MAG_ANGLE
2021-07-166.29 Addition of the following columns to the uncalibrated filter file:
XTI_PNT_JITTER (pointing jitter);
ANG_DIST_{X,Y} (angular offset in instrument X,Y coordinates)
FPM_DEADTIME (per-FPM deadtime);
MPU_LOWMEM_{FIFO,SCI}_DELTA (indication of on-board data overflow);
HV_ON (indicates if bias voltage is enabled)
FPM_SLOW_LLD (current value of low energy threshold setting);
MPU_NOISE20_COUNT (counts below 200 eV);
MEDIAN_UNDERONLY_COUNT (undershoot count rate, median of array);
BETA_ANGLE (orbital beta angle);
TIME_SINCE_SUNSET (time in seconds since beginning of orbit night);
AE8MIN,AP8MIN - AE8/AP8 modeled electron/proton fluxes
2022-10-206.31 Addition of the following columns("NICERV4"):
MPU_XRAY_PI_0030_0070 (0.3-0.7 keV rate in each detector, used to find noise)
Renaming SOLARPHI to SOLAR_PHI Removal of the following columns("NICERV4"):
{MPU,TOT}_{XRAY,OVER,UNDER,ALL}_COUNT (these counts not useful)
MPU_{FT,NOISE25}_{PI,PI_FAST}_{AVG,ERR} - (not used by downstream software and costly to compute)
The 3C50-related columns FPM_RATIO_REJ_COUNT and MPU_NOISE20_COUNT are now always computed.
2023-07-116.32 Addition of the following columns("NICERV5"):
{FPM,MPU}_NOISERING_COUNTS - tabulation of potential "noise ringer" events
SUN_BODY_AZIMUTH - the azimuth or clock angle of sun in NICER body coordinates
TOT_LOWMEM_{SCI,FIFO} - total array counts to diagnose lost events
DELTA_SLOW_LLD - array-averaged low energy threshold changes
Changes to the following columns:
MPU_LOWMEM_{FIFO,SCI}_DELTA - are upgraded to 16-bit storage type

Note that it is possible to use the newest software to re-run the associated NICER tasks and generate a most up-to-date version of the calibrated filter file.

Dealing with Filter Files from Many Observations

It is common for requested NICER observational visits to span many segments. Any requested observational visit that exceeds one calendar day is highly likely to be split into multiple segments. In addition, a long-term monitoring campaign of a target will also consist of many visits, by definition. Therefore, scientists will naturally want to deal with filter files from many observational datasets as well.

If all filter files are of the same format, this will be fine. Scientists can merge many filter files using ftmerge without problem. However, if multiple filter files have different numbers of columns, the merging task ftmerge will refuse to combine them.

How to handle this? The NICER team recommends the following strategies.

  • Use nimkfmerge. Introduced in HEASoft 6.29 (NICERDAS 8), the nimkfmerge task will readily combine filter files processed by different software versions. If nimkfmerge is set to work on filter files with mismatched columns, the rows with missing columns will be filled with NULL values.
  • Run nicerl2. The nicerl2 task will re-calibrate all observations in addition to generating the most up-to-date version of the filter file. This is a more time-consuming task, but it is more fool-proof and most likely to produce correct results.
  • Use ftmerge.Standard tools like ftmerge can be used to merge FITS table files. However, since nimkfmerge is more robust, including handling time ordering and updating various metadata keywords, we don't normally recommend using ftmerge any longer.

Merging Filter Files With 'nimkfmerge'

Merging filter files is straightforward with the tool 'nimkfmerge'. It's operation is similar to 'ftmerge' but handles more cases. However, please consider the tool 'niobsmerge' (available in HEASoft 6.32, July 2023), which also merge other observation data products at the same time. Internally, niobsmerge runs nimkfmerge for you.

While nimkfmerge is a useful tool, consider combining all observation products with the niobsmerge tool instead.

Still, if you insist on using nimkfmerge, here is how.

First, create a list file containing the names of filter files that you wish to merge. We can use the 'ls' command to do this. ls */auxil/ni*.mkf > filter.lis Here we have used the wildcard pattern */auxil/ni*.mkf to select filter files inside the observation directories in their standard places. You can tailor the wildcard pattern to match the observations or files that you want.

For example, if the filter files are gzip compressed, then you can add a .gz to the end of the wildcard patter. Or, if you only want to merge certain observations, say all observations beginning with 12345678, you can use a directory wildcard pattern such as 12345678?? (where the ? is a wildcard pattern that matches a single digit).

Next, we use the 'nimkfmerge' to combine the filter files: nimkfmerge infiles=@filter.lis outfile=merged.mkf clobber=YES where

  • infiles is the name of the list file that we just created
  • outfile is the name of the resulting merged filter file

The result is a new FITS table file, merged.mkf, which contains the combination of all input tables. Note that the input file is time-sorted by nimkfmerge.

Next Steps

You can use a time selection tool such as nimaketime on the resulting merged filter file. The result will be a GTI file that applies to the entire combined data set.

Modifications

  • 2020-05-06 - initial draft
  • 2021-04-16 - add navigation bar
  • 2021-07-16 - add HEASoft 6.29 (NICERDAS 8) changes
  • 2022-10-17 - add HEASoft 6.30 (NICERDAS 10) changes
  • 2023-07-11 - add HEASoft 6.32 (NICERDAS 11) changes; tutorial on using nimkfmerge