NAME
nicerl2 - Apply NICER Level2 standard calibration and filtering
USAGE
nicerl2 indir ...
DESCRIPTION
The nicerl2 task applies standard NICER-recommended calibration
processes to an entire NICER observation, as well as standard
screening.
This is a high level task that calls multiple sub-tasks as a
convenience to the user. It performs the following standard "Level2"
processing steps.
- nicercal - apply standard NICER calibration
- niprefilter2 - derive calibrated filter (MKF) file
- nimaketime - create standard screening good time intervals
- nicermergeclean - combine per-MPU data and filter/screen
Please see the help files for these individual tasks for more
information about what they do and how they can be used.
A user wishing to have more control over the analysis process may call
these tasks individually. nicerl2 simply rolls these subtasks up into
a single, easy-to-run task. Many, but not all, parameters of the
individual subtasks are available for nicerl2, sometimes with
different parameter names to avoid conflicts.
Unlike most HEASoft tasks, this task is designed to work on an entire
observation directory. It is expected that the input adheres to the
standard NICER directory layout: uncalibrated event files in
$INDIR/xti/event_uf; MPU housekeeping data in $INDIR/xti/hk; and
auxiliary files in $INDIR/auxil/.
However, an advanced programmer can adjust these locations with the
hidden parameters ufdir, cldir, ufafile, clfile, and mkfile. When
using these parameters, note that the certain patterns can be used as
shortcuts: "$CLDIR" for the output directory name, "$OBSID" for the
observation ID number, and "$INDIR" for the input directory. Note
that these patterns should appear explicitly with the dollar sign
included; typically the command shell will interpret '$name' as a
shell variable so appropriate quoting will be required.
OUTPUTS
Output files are placed in the directory specified by cldir,
typically $INDIR/xti/event_cl. The output products are:
- Per-module "ufa" file; lightly screened and calibrated event file for each MPU;
- Master "ufa" file; lightly screened and calibrated event file for entire array;
- Master "cl" file; screened and calibrated event file based on nimaketime screening;
- Revised "mkf" file; updated using niprefilter2 with calibrations applied.
Normally the "cl" file would be used in further analysis.
Note that the original unfiltered event files ("uf" files) are not modified.
FILTERING EVENT FILES BY TIME OR DETECTOR
nicerl2 provides various options for filtering or selecting data. This can
be roughly divided into: filtering by time, and filtering by detector. Each of
these will be discussed more below.
Filtering NICER events by time is done to screen out known "bad" times
that may include undesireable data. Values that can be screened are
stored in NICER's filter file, which is described more below. These
quantities can be divided into several categories. These categories are
listed below, along with the parameters that govern them.
- Orbit-related (nicersaafilt, saafilt)
- On-target tracking (trackfilt, ang_dist, elv, br_earth, st_valid)
- Background related (cor_range, overonly_range, overonly_expr, trumpetfilt)
- Detector related (min_fpm, underonly_range)
More information about each of these parameters can be found below, as
well as in the help for nimaketime. In
addition, the user can enter a custom "maketime" expression using the
nimaketime_gtiexpr parameter, or by entering their own custom GTI
files explicitly using the gtifiles parameter.
Users may also select or deselect certain detectors. This choice may
be based on knowledge of particular detectors' undesirable behaviors.
This selection is handled by the "detlist" parameter. detlist allows
to select or deselect certain detectors for the entire observation, as
well as for a specific time range. The default value, detlist=launch,
indicates to include all detectors enabled at launch. How this works
is described in more detail in the help file for
the nifpmsel task.
The nifpmsel task performs these detector selections and stores
per-detector exposure information in the output event file. This
information is known as FPM Selection information, and is stored in an
extension named FPM_SEL. This extension, as well as various
additional GTI_* extensions also stored in the file, provide complete
exposure information for each detector. The FPM_SEL file also
contains a columned named FPM_SEL, which is a second-by-second tally
of the exposure of each detector, and this information can be used by
the response tasks, nicerarf
and nicerrmf, to obtain accurate ARF and
RMF files.
The user may wish to run the standard processing first, with
detlist=launch, to obtain an event file with all detectors, and then
examine the resulting output for noisy or mis-behaving detectors. If
such detectors are found, it is not necessary to re-run nicerl2 from
the start again. Instead, use nifpmsel to select or deselect the
desired detectors and make a new screened event file. It is also
possible to run nicerl2 once with the default detlist=launch, and then
a second time with tasks=SCREEN and detlist the desired screening.
The result will be that the "ufa" file contains all launch detectors,
but the "cl" file contains only the selected/deselected detectors.
FPM Selection information is stored in the output event file. As
long as any event files with FPM Selection information are manipulated
by the NICER tasks nifpmsel and niextract-events, this information
will be carried along and updated for the output. However, if the
user uses tasks outside of the NICER tasks to manipulate event files,
the FPM Selection information may be lost or become inaccurate. Thus,
it is recommended to stay with the NICER tasks for merging, screening
and selecting/deselecting detectors from event files.
SELECTING FILTER FILE COLUMNS
A key output of any run of the NICER pipeline is the so-called filter file
(or .mkf file). This file is produced in the original NICER pipeline run,
but it is updated and augmented by nicerl2 using various NICER tasks.
The filter file contains many columns which are useful for diagnosing
and screening nicer data. The columns include various orbital, pointing,
and housekeeping data points.
The output of nicerl2 can be tailored to specific needs by requesting
certain columns. Complete documentation for these different columns
cannot be reproduced here. Instead, detailed information about which
columns can be produced are documented in the help files
for niprefilter
and niprefilter2.html.
Since there are several ways in which filter file columns are
calculated, requesting them can be confusing. In versions of nicerl2
v1.13 and before, the analyst had to juggle the settings for
prefilter_columns, niprefilter2_coltypes and geomag_columns. In
versions of nicerl2 v1.14, these requests are centralized in a single
parameter called "filtcolumns." The nicerl2 task automatically
populates the specific parameters as needed. Thus, it is recommended
to use the filtcolumns parameter when running nicerl2. However, for
backward compatibility, the user can specify the other parameters
individually and override the automatic settings.
Here is a summary of most commonly-used settings:
- filtcolumns=NICERV3 contains the most up-to-date
recommended filter file columns, and is recommended. It selects
the following settings,
- prefilter_columns=NICERV3
- niprefilter2_coltypes=NICERV3
- geomag_columns=kp_potsdam.fits(KP),solarphi_oulu.fits(SOLARPHI),COR_NYM
(but only if geomag_path is set to a directory path)
- filtcolumns=NICERV3,3C50 is the same as just using NICERV3,
but adds the Remillard et al. 3C50-recommended background columns.
- prefilter_columns=NICERV3
- niprefilter2_coltypes=NICERV3,3C50
- geomag_columns=kp_potsdam.fits(KP),solarphi_oulu.fits(SOLARPHI),COR_NYM
(but only if geomag_path is set to a directory path)
- filtcolumns=NICERV2 is the recommended settings as of HeaSOFT 6.28.x,
- prefilter_columns=NICERV2
- niprefilter2_coltypes=NICERV2
- geomag_columns=kp_potsdam.fits(KP)
(but only if geomag_path is set to a directory path)
- filtcolumns=NICERV1 is the recommended settings as of HeaSOFT 6.27.
- prefilter_columns=NICERV1
- niprefilter2_coltypes=BASE
- geomag_columns=NONE
filtcolumns is a comma-separated list. You can add any prefilter column name to the
list, as well as add any geomagterp column name, or any allowed prefilter2 coltypes
entry to the list, by adding a comma, followed by the requested name.
SELECTING WHICH TASKS THAT NICERL2 PERFORMS
By default, nicerl2 does several major operations as described above.
Each of these steps requires significant processing to achieve.
It can be a time consuming process to reprocess large amounts of data,
especially if only small changes are made to the inputs.
nicerl2 v1.15 and later supports a new parameter called "tasks" which
can give the analyst more fine-grained control over which tasks the
pipeline performs. These tasks are largely separable, which means
that the user can run the time-consuming processes once, and then for
small variations or iterations, only run the needed tasks. The
"tasks" parameter is a comma-separated list of task items, which can
be any one of the following,
- ALL - process all of the tasks listed below;
- CALMERGE - apply calibrations and merge the per-MPU event lists into
a single merged ufa file (nicercal and nimpumerge steps);
- MKF - create updated filter file (niprefilter and niprefilter2 steps);
- SCREEN - create a screened event list (nimaketime and nicermergeclean steps);
Although the items can be listed in any order in the parameter, they
are performed in the order described above.
Here is a strategy to save processing time by computing calibrations
and the MKF file once, followed by several different screening
variations. First, run nicerl2 to do the first two steps,
nicerl2 NNNNNNNNNN clobber=YES tasks=CALMERGE,MKF
After this run, updated filter file and event files are present in the
locations described above, but the screening steps have not yet been
performed. Now, run nicerl2 several different times with different
screening settings,
nicerl2 NNNNNNNNNN clobber=YES tasks=SCREEN overonly_range=0-2.0
... examine output ...
nicerl2 NNNNNNNNNN clobber=YES tasks=SCREEN overonly_range=0-1.5
... examine output ...
nicerl2 NNNNNNNNNN clobber=YES tasks=SCREEN overonly_range=0-1.0
In this example, different overonly_range parameter settings were
attempted, but any of the screening-related settings could be varied
at this step. The key point is that since tasks=SCREEN, the calibration
and MKF steps are not performed each time.
Of course, if the analyst wishes to vary a filter-file-related setting,
then one must also at least add MKF to the lists of tasks to perform.
Otherwise, the requested changes will not take effect. Thus, while
the tasks parameter can save significant amounts of processing time,
it also requires more logistics to keep track of which parts of the
tasks have been performed or need to be performed, depending on the
desired parameter settings.
DESTRUCTIVE OVERWRITING and WORKING WITH READ-ONLY INPUT DATA
Note that this task in its default operation will attempt to write to
the directory specified by cldir, typically $INDIR/xti/event_cl (the
"$INDIR" is a pattern replaced by the actual input directory as
described above). In addition, the filter file (.mkf file, typically
in $INDIR/auxil) is modified.
Any pre-existing files in these areas will not be destructively
overwritten unless clobber=YES (the default is clobber=NO). This is a
reminder to the user that destructive removal of data is a possibility
and that the user must explicitly allow it by setting clobber=YES.
For most typical applications, where the user is working in an
existing NICER observation directory with write access, clobber=YES
should be used on the command line to allow nicerl2 to overwrite
existing data, with the understanding that all pre-existing contents of 'cldir'
will likely be destroyed.
In some cases users may be working with a read-only archive copy of
NICER observation data, with no possibility of modifying the data in
place. In that case, even if clobber=YES is used, nicerl2 will be
unable to run using the defaults. Another slightly similar use case would be
if the user is trying different (or new) calibrations and wishes to preserve
the pre-existing calibrated outputs.
To accomodate these cases, the user must still have a
writable output directory, which can be kept separate from the input
directory. Let's say that indir=/archive/1000010101 and the output is
placed in /writable/1000010101-output.
The steps to achieve this are:
- Make writable output directory /writable/1000010101-output
- Make working copy of .mkf file in output directory
- Run nicerl2 with mkfile and cldir options
Here are the commands to be used in this case.
mkdir /writable/1000010101-output
cp -p /archive/1000010101/auxil/ni*.mkf* /writable/1000010101-output/
nicerl2 indir=/archive/1000010101 \
cldir=/writable/1000010101-output \
mkfile='$CLDIR/ni$OBSID.mkf'
PARAMETERS
- indir [filename]
- Input directory name. The directory should be a single NICER observation
directory, which in turn contains xti/{events_uf,events_cl,hk,auxil} subdirectories.
- ufdir = "$INDIR/xti/event_uf" [filename]
- Subdirectory containing per-MPU "uf" event files, which are the input to the task.
This parameter is subject to pattern substitutions $INDIR and $CLDIR as described above.
- cldir = "$INDIR/xti/event_cl" [filename]
- Subdirectory where the output files as listed above are placed.
This parameter is subject to pattern substitution $INDIR as described above.
- ufafile = "$CLDIR/ni$OBSID_0mpu7_ufa.evt" [filename]
- Name of the master "ufa" event file to be created.
This parameter is subject to pattern substitutions $INDIR, $CLDIR and $OBSID as described above.
- clfile = "$CLDIR/ni$OBSID_0mpu7_cl.evt" [filename]
- Name of the master "cl" event file to be created.
This parameter is subject to pattern substitutions $INDIR, $CLDIR and $OBSID as described above.
- mkfile = "$INDIR/auxil/ni$OBSID.mkf" [filename]
- Name of the MKF filter file, to be used as input.
This parameter is subject to pattern substitutions $INDIR, $CLDIR and $OBSID as described above.
- (mpulist = "0-6") [string]
- List of MPUs to process, either as a comma-seprated list, or as a
hypenated range. For example, "0,1,2,3,4,5,6" is the same as "0-6".
NOTE that this parameter is deprecated and will not in fact produce
the desired results of including or excluding certain MPUs from data.
Use detlist instead.
- gzip_thresh=50 [integer]
- Optimization setting. If the observation contains at least one
gzipped file larger than gzip_thresh [in megabytes], the files will
be temporarily unzipped.
- detlist="launch" [string]
- Detector selection expression, as described in the documentation
for nifpmsel.
- (tasks = "ALL") [string]
- Which task operations to perform, one of CALMERGE, MKF, SCREEN,
or ALL. See above for more details.
- (filtcolumns = "NICERV3") [string]
- As described above, filtcolumns is a comma-separated list of
columns to include in the NICER filter file. When using this keyword,
the task automatically determines which items correspond to
prefilter_columns, niprefilter2_coltypes and geomag_columns. In addition,
the shorthand notation "NICERV1", "NICERV2" and "NICERV3".
- (picalfile="CALDB") [filename]
- Name of PI gain calibration data file for slow chain pulse
heights, or "CALDB". For standard operation, the NICER team
recommends to use "CALDB".
- (pifastcalfile="CALDB") [filename]
- Name of PI_FAST gain calibration data file for fast chain pulse
heights, or "CALDB". For standard operation, the NICER team
recommends to use "CALDB".
- (timebiascalfile="CALDB") [filename]
- Name of clock calibration file, or "CALDB". For standard
operation, the NICER team recommends to use "CALDB". A value of
"NONE" disables clock calibration.
- (leapinit="AUTO") [string]
- Initialize LEAPINIT keyword of event extensions? An integer value
indicates to use that many leap seconds. A value of "AUTO" indicates use
the appropriate number of leap seconds for the given observation TSTART.
A value of "NONE" indicates to not change LEAPINIT.
- (nicercal_filtexpr="EVENT_FLAGS=bxxxx00") [string]
- Extractor filtering expression to be used for initial screening of
events. Typically this screening expression will remove over-shoot
and under-shoot events, which cannot be calibrated, but keep X-ray
events and forced triggers (EVENT_FLAGS=bxxxx00).
- Unix wildcard matching expression to locate uncalibrated
(unfiltered) events within the input directory. The default
expression will locate standard NICER unfiltered ("uf") files within
the xti/event_uf subdirectory. If the user has modified the directory
layout this pattern can be used to point nicercal to the relevant
files.
The pattern "$INDIR" is replaced by the 'indir' parameter value (note that
proper quoting is required by most command shells to avoid variable expansion).
- (hkpat = "$INDIR/xti/hk/ni??????????_?mpu?.hk{,.gz}") [string]
- Unix wildcard matching expression to locate MPU housekeeping files
within the input directory. The default expression will locate
standard NICER housekeeping files within the xti/hk subdirectory. If
the user has modified the directory layout this pattern can be used to
point nicercal to the relevant files.
The pattern "$INDIR" is replaced by the 'indir' parameter value (note that
proper quoting is required by most command shells to avoid variable expansion).
- (calstation="FLIGHT") [string]
-
Station where data was collected. For in-flight operations use
calstation=FLIGHT. For ground-based testing, other values can be
used. This parameter selects the NICSTATN keyword value in CALDB.
- (niprefilter=YES) [boolean]
-
Run niprefilter to update the filter file? (IGNORED; use tasks=MKF instead)
- (orbfile=$INDIR/auxil/ni$OBSID.orb) [string]
-
Name of the orbit file used by niprefilter.
Both the orbfile and orbfile.gz patterns are checked, in that order.
This parameter is subject to pattern substitutions $INDIR, $CLDIR and $OBSID as described above.
- (attfile=$INDIR/auxil/ni$OBSID.att) [string]
-
Name of the attitude file used by niprefilter.
Both the attfile and attfile.gz patterns are checked, in that order.
This parameter is subject to pattern substitutions $INDIR, $CLDIR and $OBSID as described above.
- (prefilter_columns=DEFAULT) [string]
-
A list of prefilter columns to compute, as a space-separated list. Please
see the documentation of prefilter, parameter
columns, for what values are allowed. A value of "DEFAULT" indicates to use
the niprefilter task default. A value of "FILTCOLUMNS" indicates to defer
to the filtcolumns parameter.
- (niprefilter2=YES) [boolean]
-
Run niprefilter2 to update the filter file? (IGNORED: use tasks=MKF instead)
- (saaregfile="CALDB") [string]
-
For niprefilter2, the name of the NICER South Atlantic Anomaly (SAA) region file, which
will be used to replace the existint NICER_SAA column. Use "CALDB" to
query CALDB. Use "NONE" to keep the existing value unchanged.
- (vehiclefile="CALDB") [string]
- For niprefilter2, retrieve International Space Station vehicle docking information as a
function of time. If set to "NONE" then this is disabled. If set to
"CALDB" or "CALDB:vehiclename", then use CALDB information for the specified
vehicle (default is "SOYUZ").
- (issmanfile="CALDB") [string]
- For niprefilter2, retrieve International Space Station maneuver information as a
function of time. If set to "NONE" then this is disabled. If set to
"CALDB" then use CALDB information.
- (robofile="CALDB") [string]
- For niprefilter2, retrieve robotic "robo" activity information as a
function of time. If set to "NONE" then this is disabled. If set to
"CALDB" then use CALDB information.
- (niprefilter2_coltypes="FILTCOLUMNS") [string]
- For niprefilter2, specify which categories of columns to produce.
See the documentation of niprefilter2
for more information. Use "DEFAULT" for the niprefilter2 task default,
and "FILTCOLUMNS" (recommended) to defer to the filtcolumns parameter.
- (geomag_columns="DEFAULT") [string]
- For niprefilter2, the name of files containing geomagnetic quantities. More than
one file can be specified using a comma-separated list. For
each file on the list, the geomag_path is prepended. Use
the notation "filename(COLNAME)" to specify that the output
column should be named COLNAME. A value of NONE means no extra
geomagnetic quantities, and a value of "DEFAULT" means to use
the task default of niprefilter2. A value of "FILTCOLUMNS" means
to defer to the filtcolumns parameter (recommended).
- (geomag_path="DEFAULT") [string]
- For niprefilter2, name of the base path for all geomagnetic files. This
path is prepended to each item in the geomag_columns list.
If DEFAULT, then the task default of geomagterp is used.
- (nicersaafilt = "YES") [boolean]
- For nimaketime, apply NICER_SAA filtering? .
- (saafilt = "NO") [boolean]
- For nimaketime, apply SAA filtering?
- (trackfilt = "YES") [boolean]
- For nimaketime, apply pointing tracking filtering?
- (st_valid = "YES") [boolean]
- For nimaketime, require star tracker valid solution?
- (ang_dist=0.015) [real]
- For nimaketime, apply ANG_DIST filtering (degrees). ANG_DIST must be less than this value.
- (elv=15) [real]
- For nimaketime, apply ELV filtering (degrees). ELV must be greater than this value.
- (br_earth=30) [real]
- For nimaketime, apply BR_EARTH filtering (degrees). BR_EARTH must be greater than this value.
- (cor_range="*-*") [string]
- For nimaketime, apply COR_SAX filtering (GeV/c). Specify a range A-B.
- (underonly_range="0-500") [string]
- For nimaketime, apply undershoot count rate range per module. Specify a range A-B.
- (overonly_range="0-1.5") [string]
- For nimaketime, apply overshoot count rate range per module. Specify a range A-B.
- (overonly_expr="1.52*OVERONLY_NORM*COR_SAX**(-0.633)") [string]
- For nimaketime, apply curve-base overshoot filtering as an expression. The
maximum value of overonly_range is substituted for OVERONOLY_NORM (or 1.0 if
no maximum is specified).
- (min_fpm=7) [integer]
- For nimaketime, apply NUM_FPM_ON filtering.
- (mpugtimerge="OR") [string]
- How to merge per-MPU GTIs into a single GTI. Either AND or OR.
For versions of NICER software before June 2021, "AND" was the
implicit default; for later versions "OR" is the default.
- (gtifiles="NONE") [string]
- Apply additional GTI filters. If you have done external filtering and
produced a GTI file, you can merge with the nimaketime GTI. Either a comma-separated
list of files or an @filelist.lis file. All GTIs (including the maketime GTI) are combined
using "AND" (intersection) merge. A value of "NONE" indicates no additional
GTI filtering.
- (pirange="20:1500") [string]
-
For nicermergeclean, pulse height screening criteria to apply. To
select PI values A to B use "A:B" notation. The asterisk can
be used to select the endpoints. For example 20:* indicates a PI
range of 20 to the maximum.
- (trumpetfilt=YES) [boolean]
-
Apply standard background PI_RATIO filtering.
-
(nicerclean_args="NONE") [string]
-
Any additional arguments to nicerclean, such as "EVENT_FLAGS=bx1x000
fastsig=250.0". See the help file for nicer clean for more
information. The gtifile, trumpetfilt, and pirange parameters are
automatically passed and there is no need to specify them again with
nicerclean_args.
- (cleanup_ufa_files="NO") [boolean]
- If yes, then clean up calibrated per-MPU "ufa" files.
- (cleanup="YES") [boolean]
- If yes, then clean up temporary files. If no, temporary files
remain. This is typically for debugging.
- (clobber = NO) [boolean]
-
If the output file already exists, then setting "clobber = yes" will cause it to be overwritten.
- (chatter = 2) [integer, 0 - 5]
- Amount of verbosity of the task. For chatter=0, no output is
printed. For chatter=5, debugging output is
printed.
- (history = YES) [boolean]
- If history = YES, then a set of HISTORY keywords will be written to
the header of the specified HDU in the output file to record the value
of all the task parameters that were used to produce the output
file.
EXAMPLES
1. Apply NICER calibration and screening to data from observation 1706221428.
nicerl2 indir=1706221428 clobber=YES
The input directory is a NICER observation number 1706221428. The
output files are placed in the observation directory under
1706221428/xti/event_cl/.
SEE ALSO
nicercal
nimaketime
niprefilter2
nicermergeclean
nifpmsel
nicerarf
nicerrmf
LAST MODIFIED
Jul 2021