Standard Pipeline Processing of an Observation with "nicerl2"
The NICER team provides a standard way to process a NICER observation. This thread describes how to perform standard processing with the task "nicerl2."
Read this thread if you want to: Perform standard analysis of a NICER observation.
Last update: 2023-08-01
The NICER team provides a way to process a NICER observation. We call that a "pipeline" processing script because it is meant to run in a standard, consistent way for every observation, with little manual intervention required.
The task nicerl2 performs the NICER team-recommended calibration and screening steps for NICER data.
The task nicerl2 is designed to perform "Level 2" analysis, which includes standard calibration, screening and filtering of data. The end result is an updated filter file (niNNNNNNNNN.mkf) as well as a cleaned event list (niNNNNNNNNNN_0mpu7_cl.evt). Here, NNNNNNNNNN is the 10-digit observational ID of the dataset (we use NNNNNNNNNN = 1234567890 below as an example).
The outputs of nicerl2 are used for subsequent analysis, such as light curve and spectral extraction. Please note that these later standard product steps are considered "Level 3" and not part of nicerl2's task list. This thread is about Level 2 only.
For more information about NICER standard processing levels, please see the NICER Processing Levels thread.
Reasons to Run nicerl2
The primary reason to run nicerl2 is to take advantage of newer NICER calibration files, or improved screening and filtering algorithms.
It is worth applying nicerl2 to any observation when you know that newer calibration is available since the last time it was processed.
When starting a project, always process NICER data using up-to-date software and calibration, even for data downloaded from the NICER archive.
Data from the NICER archive have been processed by a standard pipeline processing algorithm very similar to nicerl2. Over time, calibration and software improves, but these improvements are not regularly applied to old data in the archive. Therefore, it is always recommended to run nicerl2 on fresh data that you obtain from the NICER archive.
The same reasoning applies to standard screening criteria.
Finally, if you want to add some columns to your filter file (niNNNNNNNNNN.mkf), to use a specific background model, you may need to re-run nicerl2.
InputsThe input to "nicerl2" is a single observation dataset. As downloaded from the NICER archive, each NICER dataset is called an "observation segment" and covers no more than one calendar day.
If your scientific observation spans multiple observation segments, then you will have to run nicerl2 for each of these.
Here is what is needed:
Change into the auxil directory within the observation directory.
Run nicerl2Run nicerl2 on your observational dataset. Below is the most simple way to run it.
nicerl2 indir=1234567890 clobber=YES
When nicerl2 has completed without errors, you should find a revised filter file as well as new "ufa" and "cl"eaned event files.
Simple Scripts to Reprocess Many Observations
In many cases, users will want to reprocess many data sets at a time, rather than typing each command manually at a terminal window. We can use a simple shell (bash) script to achieve. This example, we will make a script that processes every observation in the current directory.
Prerequisite. All observation datasets should be in the
same directory. For example,
We create a script of the following form called nicer2-all.sh:
In the shell, we need to make this script executable with the Unix
Now we simply run this script:
As a variation, if you are planning to send your data to a different
directory than the observation data, you can do this as well. This
script is slightly more complicated because we have to make an output
directory for the results to be placed and copy the filter file there.
Here is an example script:
Making nicerl2 Run Faster
By default, nicerl2 runs all calibration and screening steps each time when processing data in order to obtain the most up to date and recommend processing strategies.
When starting a project, it is recommended to fully run nicerl2 to completion. While this may take a significant amount of processing time to complete, it does mean that the user can be confident that all products are up to date before proceeding with science analysis.
However, there are some times when the user may want to "reprocess" their data. Usually they want to make small variations on data selections, and don't need to re-run the entire pipeline from scratch. In that case, re-running "nicerl2" may be overkill and incur significant (and unneeded) processing overhead. The NICER software has several ways to speed up execution in this case.
The 'incremental=YES' parameter. As of HEASoft 6.31, nicerl2 has a parameter 'incremental' which causes nicerl2 to only perform some processing processing if necessary.
Specifically, before calibrating the data, nicerl2 will check any existing output files and if the calibration metadata indicates that the files have current calibration, they will be used as-is without reprocessing.
Similarly, the existing filter file is checked before processing. If it has been processed with an up-to-date version of nicerl2 (niprefilter2) with the same requested columns, regeneration of the filter file is not necessary.
Some steps, such as extracting a merged "ufa" file and screening of data, are always performed regardless of the incremental=YES setting.
If you want to force full processing, set incremental=NO, which is the default. The NICER team recommends that you do this when upgrading software or calibration, or when starting a new project.
The 'tasks' parameter. As of HEASoft 6.29, nicerl2 has a parameter called "tasks." The tasks parameter allows the user to specify which tasks to do explicitly. Essentially, while incremental=YES is automatic, the tasks parameter is a more manual way to specify what nicerl2 does. The user will need to know what steps they want to perform rather than letting nicerl2 decide.
The tasks parameter is a comma-separated list of processing items to complete. Please see the nicerl2 help file for more information but here are two common usage cases:
You can use both the 'incremental' and the 'tasks' parameters simultaneously, although it is not recommended. When you do this, nicerl2 will perform the more restrictive set of operations. For, example if you set "tasks=MKF incremental=YES", nicerl2 will skip the filter file (MKF file) generation step if the MKF file is already up to date.
The tasks parameter can be used to test different screening strategies. In a typical workflow, you would run nicerl2 once with tasks=ALL, and then run nicerl2 as many times as necessary afterward with tasks=SCREEN while experimenting with different screening techniques. Every time you run with tasks=SCREEN, nicerl2 uses the pre-existing calibrated event files and MKF files, so the screening step runs very quickly.
For example, you could first run nicerl2 once as desired
How to Prevent Writing to Observation Directory / Dealing With Read-Only Observation DataNote that this task in its default operation will attempt to write to the observational directory. Some "old" files may be altered or removed.
Any pre-existing files in these areas will not be destructively overwritten unless clobber=YES (the default is clobber=NO). This is a reminder to the user that destructive removal of data is a possibility and that the user must explicitly allow it by setting clobber=YES.
For most typical applications, where the user is working in an existing NICER observation directory with write access, clobber=YES should be used on the command line to allow nicerl2 to overwrite existing data, with the understanding that pre-existing Level 2 derived files are likely be destroyed.
In some cases users may be working with a read-only archive copy of NICER observation data, with no possibility of modifying the data in place. In that case, even if clobber=YES is used, nicerl2 will be unable to run using the defaults.
Another slightly similar use case would be if the user is trying different (or new) calibrations and wishes to preserve the pre-existing calibrated outputs.
To accomodate these cases, the user must still have a writable output directory, which can be kept separate from the input directory. Let's say that indir=/archive/1234567890 and the output is placed in /writable/1234567890-output.
The steps to achieve this are:
Here are the commands to be used in this case.
When complete, the new output files will be placed in the /writable area, and the /archive area will be left untouched.
Variations: Selecting Columns for Background Models
As of NICERDAS 10, there are three available NICER background models to choose from. These are the SCORPEON, 3C50 and Space Weather models. More information on these tools can be found on the NICER Background Estimator Tools page.
In previous versions of NICERDAS, users were required to manually add columns to the filter file in order to make background modeling work properly. For example, the Space Weather model requires the "KP" column, whereas the 3C50 model requires other specific housekeeping count rate columns.
Starting with NICERDAS 10, all of the needed columns are added by default, as long as the default filtcolumns=NICERV4 (or higher version) is used. For that reason, no special command line options are required any more to support background modeling. This is done automatically.
Since some of these columns are deal with geomagnetic quantities, you will need to download and update geomagnetic data when processing NICER data. More information is on the Geomagnetic Thread. As long as you set your GEOMAG_PATH environment variable to point to the directory where geomagnetic data is stored, nicerl2 will automatically use it.
Variations: Dealing with Data Processed with Older Versions of nicerl2Some NICER analysts have built up a large database of NICER observational datasets. They may wish to merge these datasets together to gain the power of large exposures. Over time, additional columns have been added to the filter files by the standard processing. Users who are merging may not wish that, because it is impossible to use "ftmerge" to merge filter files with different kinds of columns.
One solution to this is to simply reprocess all datasets with modern calibration and modern version of nicerl2. Please see above for this method.
Another solution is to process the "new" data with "old" columns. As of HEASoft 6.29, there is a way to do this. This version of NICER tools added some new columns to the standard filter file. In order to track this you can use the filtcolumns parameter of nicerl2.
Thus, for a user that has processed most of there data with HEASoft 6.26
or earlier, they can run nicerl2 as follows.
Next StepsThe next steps will be to use the products of nicerl2 for light curve and spectral extraction. Please see the spectral and light curve product threads for more information.