NAME

nicerl2 - Apply NICER Level2 standard calibration and filtering

USAGE

nicerl2 indir ...

DESCRIPTION

The nicerl2 task applies standard NICER-recommended calibration processes to an entire NICER observation, as well as standard screening.

This is a high level task that calls multiple sub-tasks as a convenience to the user. It performs the following standard "Level2" processing steps.

  1. nicercal - apply standard NICER calibration
  2. niprefilter2 - derive calibrated filter (MKF) file
  3. nimaketime - create standard screening good time intervals
  4. nicermergeclean - combine per-MPU data and filter/screen
Please see the help files for these individual tasks for more information about what they do and how they can be used.

A user wishing to have more control over the analysis process may call these tasks individually. nicerl2 simply rolls these subtasks up into a single, easy-to-run task. Many, but not all, parameters of the individual subtasks are available for nicerl2, sometimes with different parameter names to avoid conflicts.

Unlike most HEASoft tasks, this task is designed to work on an entire observation directory. It is expected that the input adheres to the standard NICER directory layout: uncalibrated event files in $INDIR/xti/event_uf; MPU housekeeping data in $INDIR/xti/hk; and auxiliary files in $INDIR/auxil/.

However, an advanced programmer can adjust these locations with the hidden parameters ufdir, cldir, ufafile, clfile, and mkfile. When using these parameters, note that the certain patterns can be used as shortcuts: "$CLDIR" for the output directory name, "$OBSID" for the observation ID number, and "$INDIR" for the input directory. Note that these patterns should appear explicitly with the dollar sign included; typically the command shell will interpret '$name' as a shell variable so appropriate quoting will be required.

OUTPUTS

Output files are placed in the directory specified by cldir, typically $INDIR/xti/event_cl. The output products are:

Normally the "cl" file would be used in further analysis.

Note that the original unfiltered event files ("uf" files) are not modified.

FILTERING EVENT FILES BY TIME OR DETECTOR

nicerl2 provides various options for filtering or selecting data. This can be roughly divided into: filtering by time, and filtering by detector. Each of these will be discussed more below.

Filtering NICER events by time is done to screen out known "bad" times that may include undesireable data. Values that can be screened are stored in NICER's filter file, which is described more below. These quantities can be divided into several categories. These categories are listed below, along with the parameters that govern them.

More information about each of these parameters can be found below, as well as in the help for nimaketime. In addition, the user can enter a custom "maketime" expression using the nimaketime_gtiexpr parameter, or by entering their own custom GTI files explicitly using the gtifiles parameter.

Users may also select or deselect certain detectors. This choice may be based on knowledge of particular detectors' undesirable behaviors. This selection is handled by the "detlist" parameter. detlist allows to select or deselect certain detectors for the entire observation, as well as for a specific time range. The default value, detlist=launch, indicates to include all detectors enabled at launch. How this works is described in more detail in the help file for the nifpmsel task. The nifpmsel task performs these detector selections and stores per-detector exposure information in the output event file. This information is known as FPM Selection information, and is stored in an extension named FPM_SEL. This extension, as well as various additional GTI_* extensions also stored in the file, provide complete exposure information for each detector. The FPM_SEL file also contains a columned named FPM_SEL, which is a second-by-second tally of the exposure of each detector, and this information can be used by the response tasks, nicerarf and nicerrmf, to obtain accurate ARF and RMF files.

The user may wish to run the standard processing first, with detlist=launch, to obtain an event file with all detectors, and then examine the resulting output for noisy or mis-behaving detectors. If such detectors are found, it is not necessary to re-run nicerl2 from the start again. Instead, use nifpmsel to select or deselect the desired detectors and make a new screened event file. It is also possible to run nicerl2 once with the default detlist=launch, and then a second time with tasks=SCREEN and detlist the desired screening. The result will be that the "ufa" file contains all launch detectors, but the "cl" file contains only the selected/deselected detectors.

FPM Selection information is stored in the output event file. As long as any event files with FPM Selection information are manipulated by the NICER tasks nifpmsel and niextract-events, this information will be carried along and updated for the output. However, if the user uses tasks outside of the NICER tasks to manipulate event files, the FPM Selection information may be lost or become inaccurate. Thus, it is recommended to stay with the NICER tasks for merging, screening and selecting/deselecting detectors from event files.

SELECTING FILTER FILE COLUMNS

A key output of any run of the NICER pipeline is the so-called filter file (or .mkf file). This file is produced in the original NICER pipeline run, but it is updated and augmented by nicerl2 using various NICER tasks. The filter file contains many columns which are useful for diagnosing and screening nicer data. The columns include various orbital, pointing, and housekeeping data points.

The output of nicerl2 can be tailored to specific needs by requesting certain columns. Complete documentation for these different columns cannot be reproduced here. Instead, detailed information about which columns can be produced are documented in the help files for niprefilter and niprefilter2.html.

Since there are several ways in which filter file columns are calculated, requesting them can be confusing. In versions of nicerl2 v1.13 and before, the analyst had to juggle the settings for prefilter_columns, niprefilter2_coltypes and geomag_columns. In versions of nicerl2 v1.14, these requests are centralized in a single parameter called "filtcolumns." The nicerl2 task automatically populates the specific parameters as needed. Thus, it is recommended to use the filtcolumns parameter when running nicerl2. However, for backward compatibility, the user can specify the other parameters individually and override the automatic settings.

Here is a summary of most commonly-used settings:

filtcolumns is a comma-separated list. You can add any prefilter column name to the list, as well as add any geomagterp column name, or any allowed prefilter2 coltypes entry to the list, by adding a comma, followed by the requested name.

SELECTING WHICH TASKS THAT NICERL2 PERFORMS

By default, nicerl2 does several major operations as described above. Each of these steps requires significant processing to achieve. It can be a time consuming process to reprocess large amounts of data, especially if only small changes are made to the inputs.

nicerl2 v1.15 and later supports a new parameter called "tasks" which can give the analyst more fine-grained control over which tasks the pipeline performs. These tasks are largely separable, which means that the user can run the time-consuming processes once, and then for small variations or iterations, only run the needed tasks. The "tasks" parameter is a comma-separated list of task items, which can be any one of the following,

Although the items can be listed in any order in the parameter, they are performed in the order described above.

Here is a strategy to save processing time by computing calibrations and the MKF file once, followed by several different screening variations. First, run nicerl2 to do the first two steps, 

   nicerl2 NNNNNNNNNN clobber=YES tasks=CALMERGE,MKF
After this run, updated filter file and event files are present in the locations described above, but the screening steps have not yet been performed. Now, run nicerl2 several different times with different screening settings, 
   nicerl2 NNNNNNNNNN clobber=YES tasks=SCREEN overonly_range=0-2.0 
   ... examine output ...
   nicerl2 NNNNNNNNNN clobber=YES tasks=SCREEN overonly_range=0-1.5
   ... examine output ...
   nicerl2 NNNNNNNNNN clobber=YES tasks=SCREEN overonly_range=0-1.0
In this example, different overonly_range parameter settings were attempted, but any of the screening-related settings could be varied at this step. The key point is that since tasks=SCREEN, the calibration and MKF steps are not performed each time.

Of course, if the analyst wishes to vary a filter-file-related setting, then one must also at least add MKF to the lists of tasks to perform. Otherwise, the requested changes will not take effect. Thus, while the tasks parameter can save significant amounts of processing time, it also requires more logistics to keep track of which parts of the tasks have been performed or need to be performed, depending on the desired parameter settings.

DESTRUCTIVE OVERWRITING and WORKING WITH READ-ONLY INPUT DATA

Note that this task in its default operation will attempt to write to the directory specified by cldir, typically $INDIR/xti/event_cl (the "$INDIR" is a pattern replaced by the actual input directory as described above). In addition, the filter file (.mkf file, typically in $INDIR/auxil) is modified.

Any pre-existing files in these areas will not be destructively overwritten unless clobber=YES (the default is clobber=NO). This is a reminder to the user that destructive removal of data is a possibility and that the user must explicitly allow it by setting clobber=YES.

For most typical applications, where the user is working in an existing NICER observation directory with write access, clobber=YES should be used on the command line to allow nicerl2 to overwrite existing data, with the understanding that all pre-existing contents of 'cldir' will likely be destroyed.

In some cases users may be working with a read-only archive copy of NICER observation data, with no possibility of modifying the data in place. In that case, even if clobber=YES is used, nicerl2 will be unable to run using the defaults. Another slightly similar use case would be if the user is trying different (or new) calibrations and wishes to preserve the pre-existing calibrated outputs.

To accomodate these cases, the user must still have a writable output directory, which can be kept separate from the input directory. Let's say that indir=/archive/1000010101 and the output is placed in /writable/1000010101-output.

The steps to achieve this are:

Here are the commands to be used in this case. 
     mkdir /writable/1000010101-output
     cp -p /archive/1000010101/auxil/ni*.mkf* /writable/1000010101-output/
     nicerl2 indir=/archive/1000010101 \
         cldir=/writable/1000010101-output \
         mkfile='$CLDIR/ni$OBSID.mkf'

PARAMETERS

indir [filename]
Input directory name. The directory should be a single NICER observation directory, which in turn contains xti/{events_uf,events_cl,hk,auxil} subdirectories.
ufdir = "$INDIR/xti/event_uf" [filename]
Subdirectory containing per-MPU "uf" event files, which are the input to the task. This parameter is subject to pattern substitutions $INDIR and $CLDIR as described above.
cldir = "$INDIR/xti/event_cl" [filename]
Subdirectory where the output files as listed above are placed. This parameter is subject to pattern substitution $INDIR as described above.
ufafile = "$CLDIR/ni$OBSID_0mpu7_ufa.evt" [filename]
Name of the master "ufa" event file to be created. This parameter is subject to pattern substitutions $INDIR, $CLDIR and $OBSID as described above.
clfile = "$CLDIR/ni$OBSID_0mpu7_cl.evt" [filename]
Name of the master "cl" event file to be created. This parameter is subject to pattern substitutions $INDIR, $CLDIR and $OBSID as described above.
mkfile = "$INDIR/auxil/ni$OBSID.mkf" [filename]
Name of the MKF filter file, to be used as input. This parameter is subject to pattern substitutions $INDIR, $CLDIR and $OBSID as described above.
(mpulist = "0-6") [string]
List of MPUs to process, either as a comma-seprated list, or as a hypenated range. For example, "0,1,2,3,4,5,6" is the same as "0-6". NOTE that this parameter is deprecated and will not in fact produce the desired results of including or excluding certain MPUs from data. Use detlist instead.
gzip_thresh=50 [integer]
Optimization setting. If the observation contains at least one gzipped file larger than gzip_thresh [in megabytes], the files will be temporarily unzipped.
detlist="launch" [string]
Detector selection expression, as described in the documentation for nifpmsel.
(tasks = "ALL") [string]
Which task operations to perform, one of CALMERGE, MKF, SCREEN, or ALL. See above for more details.
(filtcolumns = "NICERV3") [string]
As described above, filtcolumns is a comma-separated list of columns to include in the NICER filter file. When using this keyword, the task automatically determines which items correspond to prefilter_columns, niprefilter2_coltypes and geomag_columns. In addition, the shorthand notation "NICERV1", "NICERV2" and "NICERV3".
(picalfile="CALDB") [filename]
Name of PI gain calibration data file for slow chain pulse heights, or "CALDB". For standard operation, the NICER team recommends to use "CALDB".
(pifastcalfile="CALDB") [filename]
Name of PI_FAST gain calibration data file for fast chain pulse heights, or "CALDB". For standard operation, the NICER team recommends to use "CALDB".
(timebiascalfile="CALDB") [filename]
Name of clock calibration file, or "CALDB". For standard operation, the NICER team recommends to use "CALDB". A value of "NONE" disables clock calibration.
(leapinit="AUTO") [string]
Initialize LEAPINIT keyword of event extensions? An integer value indicates to use that many leap seconds. A value of "AUTO" indicates use the appropriate number of leap seconds for the given observation TSTART. A value of "NONE" indicates to not change LEAPINIT.
(nicercal_filtexpr="EVENT_FLAGS=bxxxx00") [string]
Extractor filtering expression to be used for initial screening of events. Typically this screening expression will remove over-shoot and under-shoot events, which cannot be calibrated, but keep X-ray events and forced triggers (EVENT_FLAGS=bxxxx00).
Unix wildcard matching expression to locate uncalibrated (unfiltered) events within the input directory. The default expression will locate standard NICER unfiltered ("uf") files within the xti/event_uf subdirectory. If the user has modified the directory layout this pattern can be used to point nicercal to the relevant files. The pattern "$INDIR" is replaced by the 'indir' parameter value (note that proper quoting is required by most command shells to avoid variable expansion).
(hkpat = "$INDIR/xti/hk/ni??????????_?mpu?.hk{,.gz}") [string]
Unix wildcard matching expression to locate MPU housekeeping files within the input directory. The default expression will locate standard NICER housekeeping files within the xti/hk subdirectory. If the user has modified the directory layout this pattern can be used to point nicercal to the relevant files. The pattern "$INDIR" is replaced by the 'indir' parameter value (note that proper quoting is required by most command shells to avoid variable expansion).
(calstation="FLIGHT") [string]
Station where data was collected. For in-flight operations use calstation=FLIGHT. For ground-based testing, other values can be used. This parameter selects the NICSTATN keyword value in CALDB.
(niprefilter=YES) [boolean]
Run niprefilter to update the filter file? (IGNORED; use tasks=MKF instead)
(orbfile=$INDIR/auxil/ni$OBSID.orb) [string]
Name of the orbit file used by niprefilter. Both the orbfile and orbfile.gz patterns are checked, in that order. This parameter is subject to pattern substitutions $INDIR, $CLDIR and $OBSID as described above.
(attfile=$INDIR/auxil/ni$OBSID.att) [string]
Name of the attitude file used by niprefilter. Both the attfile and attfile.gz patterns are checked, in that order. This parameter is subject to pattern substitutions $INDIR, $CLDIR and $OBSID as described above.
(prefilter_columns=DEFAULT) [string]
A list of prefilter columns to compute, as a space-separated list. Please see the documentation of prefilter, parameter columns, for what values are allowed. A value of "DEFAULT" indicates to use the niprefilter task default. A value of "FILTCOLUMNS" indicates to defer to the filtcolumns parameter.
(niprefilter2=YES) [boolean]
Run niprefilter2 to update the filter file? (IGNORED: use tasks=MKF instead)
(saaregfile="CALDB") [string]
For niprefilter2, the name of the NICER South Atlantic Anomaly (SAA) region file, which will be used to replace the existint NICER_SAA column. Use "CALDB" to query CALDB. Use "NONE" to keep the existing value unchanged.
(vehiclefile="CALDB") [string]
For niprefilter2, retrieve International Space Station vehicle docking information as a function of time. If set to "NONE" then this is disabled. If set to "CALDB" or "CALDB:vehiclename", then use CALDB information for the specified vehicle (default is "SOYUZ").
(issmanfile="CALDB") [string]
For niprefilter2, retrieve International Space Station maneuver information as a function of time. If set to "NONE" then this is disabled. If set to "CALDB" then use CALDB information.
(robofile="CALDB") [string]
For niprefilter2, retrieve robotic "robo" activity information as a function of time. If set to "NONE" then this is disabled. If set to "CALDB" then use CALDB information.
(niprefilter2_coltypes="FILTCOLUMNS") [string]
For niprefilter2, specify which categories of columns to produce. See the documentation of niprefilter2 for more information. Use "DEFAULT" for the niprefilter2 task default, and "FILTCOLUMNS" (recommended) to defer to the filtcolumns parameter.
(geomag_columns="DEFAULT") [string]
For niprefilter2, the name of files containing geomagnetic quantities. More than one file can be specified using a comma-separated list. For each file on the list, the geomag_path is prepended. Use the notation "filename(COLNAME)" to specify that the output column should be named COLNAME. A value of NONE means no extra geomagnetic quantities, and a value of "DEFAULT" means to use the task default of niprefilter2. A value of "FILTCOLUMNS" means to defer to the filtcolumns parameter (recommended).
(geomag_path="DEFAULT") [string]
For niprefilter2, name of the base path for all geomagnetic files. This path is prepended to each item in the geomag_columns list. If DEFAULT, then the task default of geomagterp is used.
(nicersaafilt = "YES") [boolean]
For nimaketime, apply NICER_SAA filtering? .
(saafilt = "NO") [boolean]
For nimaketime, apply SAA filtering?
(trackfilt = "YES") [boolean]
For nimaketime, apply pointing tracking filtering?
(st_valid = "YES") [boolean]
For nimaketime, require star tracker valid solution?
(ang_dist=0.015) [real]
For nimaketime, apply ANG_DIST filtering (degrees). ANG_DIST must be less than this value.
(elv=15) [real]
For nimaketime, apply ELV filtering (degrees). ELV must be greater than this value.
(br_earth=30) [real]
For nimaketime, apply BR_EARTH filtering (degrees). BR_EARTH must be greater than this value.
(cor_range="*-*") [string]
For nimaketime, apply COR_SAX filtering (GeV/c). Specify a range A-B.
(underonly_range="0-500") [string]
For nimaketime, apply undershoot count rate range per module. Specify a range A-B.
(overonly_range="0-1.5") [string]
For nimaketime, apply overshoot count rate range per module. Specify a range A-B.
(overonly_expr="1.52*OVERONLY_NORM*COR_SAX**(-0.633)") [string]
For nimaketime, apply curve-base overshoot filtering as an expression. The maximum value of overonly_range is substituted for OVERONOLY_NORM (or 1.0 if no maximum is specified).
(min_fpm=7) [integer]
For nimaketime, apply NUM_FPM_ON filtering.
(mpugtimerge="OR") [string]
How to merge per-MPU GTIs into a single GTI. Either AND or OR. For versions of NICER software before June 2021, "AND" was the implicit default; for later versions "OR" is the default.
(gtifiles="NONE") [string]
Apply additional GTI filters. If you have done external filtering and produced a GTI file, you can merge with the nimaketime GTI. Either a comma-separated list of files or an @filelist.lis file. All GTIs (including the maketime GTI) are combined using "AND" (intersection) merge. A value of "NONE" indicates no additional GTI filtering.
(pirange="20:1500") [string]
For nicermergeclean, pulse height screening criteria to apply. To select PI values A to B use "A:B" notation. The asterisk can be used to select the endpoints. For example 20:* indicates a PI range of 20 to the maximum.
(trumpetfilt=YES) [boolean]
Apply standard background PI_RATIO filtering.
(nicerclean_args="NONE") [string]
Any additional arguments to nicerclean, such as "EVENT_FLAGS=bx1x000 fastsig=250.0". See the help file for nicer clean for more information. The gtifile, trumpetfilt, and pirange parameters are automatically passed and there is no need to specify them again with nicerclean_args.
(cleanup_ufa_files="NO") [boolean]
If yes, then clean up calibrated per-MPU "ufa" files.
(cleanup="YES") [boolean]
If yes, then clean up temporary files. If no, temporary files remain. This is typically for debugging.
(clobber = NO) [boolean]
If the output file already exists, then setting "clobber = yes" will cause it to be overwritten.

(chatter = 2) [integer, 0 - 5]
Amount of verbosity of the task. For chatter=0, no output is printed. For chatter=5, debugging output is printed.

(history = YES) [boolean]
If history = YES, then a set of HISTORY keywords will be written to the header of the specified HDU in the output file to record the value of all the task parameters that were used to produce the output file.

EXAMPLES

1. Apply NICER calibration and screening to data from observation 1706221428. 

  nicerl2 indir=1706221428 clobber=YES

The input directory is a NICER observation number 1706221428. The output files are placed in the observation directory under 1706221428/xti/event_cl/.

SEE ALSO

nicercal

nimaketime

niprefilter2

nicermergeclean

nifpmsel

nicerarf

nicerrmf

LAST MODIFIED

Jul 2021