Complete Light Curve Product Pipeline (nicerl3-lc)

Overview

Almost all users will wish to product a light curve for their X-ray observational data. This thread describes a complete end-to-end spectral product pipeline called nicerl3-lc.

Read this thread if you want to: Extract light curve products in a straightforward way

Last update: 2023-07-11

Introduction

Light curve analysis is key to most X-ray analysis. In some cases it provides context for analysis. For example, if there are bursts or eclipses, the scientist can use an X-ray light curve to detect these occurrences. In other cases, light curve analysis is the beginning of timing analysis, such as power spectroscopy. Thus, extracting a light curve is a fundamental task that almost all NICER users will need to do.

This thread describes how to extract all NICER light curve products using a single pipeline task called nicerl3-lc. This task generates all products needed for light curve analysis: light curve and (optional) background estimate.

The nicerl3-lc task became available with HEASoft 6.31, which was released in November, 2022. The task provides a straightforward and streamlined way to extract products with NICER-recommended methods.

While nicerl3-lc is targeted at the "new user" of NICER data, it is equally usable by advanced users. Many of the complicated and/or poorly documented steps are now done automatically.


Figure 1. Workflow showing nicerl3-spect and nicerl3-lc to make standard spectral and light curve products.

Figure 1 shows the basic workflow for any NICER analysis using the new nicerl3 high level pipeline tasks. The user first runs nicerl2 to apply standard calibration and screening to the data, and then runs the nicerl3-lc task to create light curve products. For more information about the "levels" of NICER analysis, please see the NICER Processing Levels thread.

The input of nicerl3-lc is the output of nicerl2. Namely, the clean event file ("cl" file), plus other auxiliary files like the filter file (MKF file). Some background models, such as 3C50, also use the unfiltered event files ("ufa" files). Without special prompting, the nicerl3-lc task automatically recognizes the directory layout of a NICER standard observation as well the situation where the user directed nicerl2's outputs to a different directory.

The nicerl3-lc task will make many automatic decisions for you, such as choice of background model, but the advanced user is free to make alternate choices, as described below.

As output, nicerl3-lc makes the following products:

  • extracted light curve from cleaned event file (lcfile)
  • Background light curve file (bkgfile)

nicerl3-lc also creates a graphical plot of the light curve. This is suitable for quicklook pipelines that want a standard view of the light curve.

nicerl3-lc chains together multiple standard NICER tasks in the team-recommended fashion. Although not recommended, users can also run the tasks manually themselves as well. The tasks are,

Although individual manual tasks are provided, the NICER team recommends to use nicerl3-lc to generate all products in a consistent manner.

The following discussion shows the basic operation of nicerl3-lc. For more detailed information on task operation, please see the nicerl3-lc help file

Prerequisites

The nicerl3-lc task requires the output of the nicerl2 task. This means EITHER, the data processed "in-place" in an existing observation directory, or an output directory as described in the "WORKING WITH READ-ONLY INPUT DATA" section of the nicerl2 help file.

Basically, the task will look in the supplied input directory name for the subdirectory xti/event_cl, or in the subdirectory itself, for the required files. The required files are described above.

Extract Light Curve Products

Run the nicerl3-lc spectral product pipeline task with the following command.

nicerl3-lc 1234567890 pirange=300-1500 timebin=60.0 clobber=YES

where

  • "1234567890" is the name of the observation directory with cleaned and calibrated data
  • pirange is the channel (energy) range
  • timebin is the time bin size in seconds

Upon completion, nicerl3-lc will product the products as described above. They will be placed in the output directory. As an example, let's list the directory and see what products were produced by this run. ls -l 1234567890/xti/event_cl/ -rw-r--r--. 1 user user 74880 Mon 01 00:00 ni5010100101mpu7_sr.lc ;; light curve

As you can see, a light curve is produced. By default no background is produced unless explicitly requested.

Plot Light Curve with 'fplot'

We can make a quick plot of the light curve using the fplot command. fplot 1234567890/xti/event_cl/ni1234567890mpu7_sr.lc offset=YES Name of X Axis Parameter[error][] TIME Name of Y Axis Parameter[error] up to 8 allowed[] RATE[ERROR] Lists of rows[-] Device: /XWindow, /XTerm, /TK, /PS, etc[/xs] Any legal PLT command[ ]

Here we are using TIME as the X-axis, RATE as the Y-axis, and the /xs "X windows" plotting device. We use "offset=YES" so that the X-axis is numerically readable. The result is shown in Figure 2.
Figure 2. Sample nicerl3-lc light curve plot.

Of course, in this example the observation ID number is 1234567890; this will vary according to your particular observation. If you are using an alternate output directory, you may need to use 1234567890/ni1234567890_sr.lc instead.

Light Curve Normalization

As of NICERDAS 11, released in July 2023, the nicerl3-lc task will automatically normalize light curves by the number of enabled and/or selected detectors. This has the added benefit of having NICER standard light curves be more representative of the source's true flux. If the number of enabled and/or selected detectors varies during an observation those would be visible in the raw light curve as jumps, which are indicative of variations in the number of detectors rather than the source's flux.

NOTE: the renormalization capability described in this section was not performed in HEASoft releases before July 2023 (NICERDAS 10 / HEASoft 6.31.1 or earlier). For those earlier releases, the light curves were not normalized, and they contain the raw array count rate, equivalent to detnormtype=none.

How to enable. Renormalization is enabled by default in NICERDAS 11 / HEASoft 6.32 or later. Disable normalization to recover earlier behavior by setting detnormtype=none.

How light curves are normalized. By default, light curves are normalized to the count rate of an equivalent 52-detector array. For example, if 26 detectors are selected in an observation, the raw count rates are multipled by a factor of 2. The filter file and/or the FPM Selection information accompanying the data are used to determine the normalization on a second-by-second basis. Thus, observations where the number of enabled detectors varies from time to time are accomodated.

The parameter detnormtype determines the normalization factor. Set detnormtype=arr52 to normalize to an equivalent 52-detector array. Set detnormtype=fpm to normalize to an average enabled detector.

When not to normalize. There are some science cases where normalization may not be the best approach to light curve analysis. One may be to maintain compatibility to a previous analysis when this capability was not available. Another may be for sensitive timing analysis where raw counts are preferred. In these cases, set detnormtype=none.

Edge effects. NICER detector housekeeping is not 100% accurate. Detectors are not enabled immediately when commanded to start taking data: there may be some brief transient behavior. Also, detector housekeeping, which is reported once per second does not capture where in that second the on/off transition occurred. Therefore, detectors that transition between on-off states are "blanked out" for a period around the transition time. The amount of blanking is governed by the detnormbuff parameter (whose default is 1.6 seconds). Set detnormbuff=0.0 to use all time bins, including potential temporal transients during detector start-up.

Also, small changes in numbers of enabled detectors do not have this blanking. This behavior is governed by the detnormchg parameter, whose default 10 in units of 10%. In other words, for a 52-detector array, changes by 10% (5 detectors or fewer) wlil be fully included with no blanking. Set detnormchg=0.0 to blank out all detector changes.

The default normalization as of NICERDAS 11 / HEASoft 6.32 is to scale to a 52-detector array count rate. Use detnormtype=fpm to scale to per-detector rate. Disable normalization by setting detnormtype=none (which is equivalent to what happens in earlier releases).

Variation: Applying Additional Screening to Data

A common situation will be that the user desires to apply additional temporal screening to their data. The cause of this is typically scientifically-motivated. For example, a user might want to look at the spectrum of a burst, or during eclipse, or during a high or low state.

Generally speaking the user will have to construct a GTI (Good Time Interval) file which contains the desired time range. You may be able to use maketime or nimaketime to get a new GTI file, or you may have to construct a GTI file manually yourself. For an example of how to do this (for Swift) please see the Swift BAT thread Making BAT Light Curves (but please beware that for NICER MJDREFI=56658 and MJDREFF=7.775925925925930E-04).

Once you have a GTI file, you can apply it simply from the nicerl3-lc command line, nicerl3-lc 1234567890 gtifile=myfile.gti clobber=YES where myfile.gti is the name of your custom GTI file.

If you want to apply different quality screening levels, such as overshoots or undershoots, it is recommended that users run nicerl2 again. To reduce execution time, you can set the nicerl2 parameter tasks=SCREEN and it will only perform the screening step again, without performing the other more time consuming steps.

Variation: Choosing Different Background Models

THe NICERDAS suite contains multiple NICER background models, starting with NICERDAS version 10 (HEASoft 6.31, released November 2022). You can choose to enable background models when you run nicerl3-lc, but there are limitations as described below.

As of the HEASoft 6.31 release, only the Space Weather background model is supported for light curve background estimates.

Background estimates are not stored as a separate file. Rather, the light curve will contain an additional column called BACKV for the background estimate (and BACKE for the estimated background error, if any).

In addition, creation of light curve background estimate can be time consuming. For this reason, background estimates are disabled by default. However, you can enable the Space Weather background model. This is described briefly below.

Space Weather The Space Weather model due to Gendreau and Corcoran is distributed with NICERDAS. All software and calibration files are included within the NICER environment, with no separate downloads required. Settings "bkgmodeltype=sw" (for more details see nispaceweather).

For example, to run with the Space Weather model, use the following command, nicerl3-lc 1234567890 300-1500 60.0 bkgmodeltype=sw clobber=YES Please note that the execution time scales with the number of time bins and becomes less accurate when time bins are small. Therefore, it is not recommended to enable the Space Weather background for bin sizes smaller than 32 seconds.

Each of the background models has additional parameters or settings that allow more in-depth control of the model generation. nicerl3-lc does support these. For the SCORPEON model, the bkgcomponents, bkgvariant, bkgsoftlanding and bkgver parameters are passed to the SCORPEON modeling tasks, which are documented on the SCORPEON help file pages. For the 3C50 model, the hbgcut and s0cut parameters that control screening of extreme conditions are passed to the nibackgen3C50 model, which are documented on the nibackgen3C50 page. For the Space Weather model, there are no additional parameters.

Variation: Changing Energy Range or Time Bin Size

Users will naturally want to choose the time binning and energy range that is right for their scientific project.

The energy range is given by the "pirange" parameter, which has no default. The user must specify an energy range as a PI channel range "start-stop". Each NICER PI bin is 10 eV (see Gain Calibration for more details), so a PI range of 300-1500 is equivalent to 3.0-15.0 keV.

The time bin size must be given in units of seconds.

As an example, if the scientist wishes a 0.3-2.0 keV light curve with 100 second bins, they could use the following command: nicerl3-lc 1234567890 30-200 100.0 clobber=YES

Variation: Choosing Different Graphical Output Options

By default, the nicerl3-lc task produces a PNG output file. The user can make other plotting choices.

To completely disable an output plot, use the doplot=NO option.

To choose different graphical output formats, use the plotfiles parameter. For example, to output both postscript and PNG files, use the following command line. nicerl3-lc 1234567890 30-200 60.0 plotfiles="myfile.ps/cps myfile.png/png" Here, "/cps" indicates Color PostScript. Another possible option is GIF, such as nicerl3-lc 1234567890 30-200 60.0 plotfiles=myfile.gif/gif Any file name/device name combination can be given, in the style of XSPEC's "hard" command.

Please note that the default graphical output format is PNG. If your computer does not have a PNG graphical library installed (typically "libpng"), you may experience an error. If you can, you may wish to install a PNG output library or driver for your computer, or you may wish to try an olternative graphical output as described above.

By default the X-axis is TIME. However, in some cases, it may be difficult to view the details of such a light curve because NICER's observations are often temporarlly disjoint. To plot versus sample number instead of time, set plottime=sandwich, and then all samples will be sandwiched together.

Variation: Choosing Different File Names

nicerl3-lc will choose sensible file names for its output files, but you can modify this.

By default, nicerl3-lc will notice the name of the cleaned event file and use that file name and subdirectory as a basis for all output file names. For example, if the cleaned event file name is 1234567890/xti/event_cl/ni1234567890_0mpu7_cl.evt nicerl3-lc will remove the the "_0mpu7_cl.evt" portion and use the remaining portion as the root name for all outputs. For example the output light curve will be named, 1234567890/xti/event_cl/ni1234567890mpu7_sr.lc and so on. The "mpu7_sr.lc" gets added by nicerl3-lc.

If you are trying different variations with the same basic inputs, consider setting the suffix parameter to avoid overwriting existing files. For example, if you are comparing different energy ranges with the same inputs, you could run nicerl3-lc twice with two different suffix values. In this case the following commands would achieve this, nicerl3-lc 1234567890 30-200 100.0 suffix=soft clobber=YES nicerl3-lc 1234567890 200-800 100.0 suffix=hard clobber=YES And the resulting output files will have either "soft" or "hard" as file name suffices for the 0.3-2.0 and 2.0-8.0 keV energy ranges, respectively.

If you want to take complete control of the process, you can set the output file name options explicitly to what you desire. Please see the nicerl3-lc help file for more information.

What Can Go Wrong: No Good Time

Generally speaking, nicerl3-lc can terminate due to an error condition. Please read the error message for more information about how to remedy the situation.

However, there one special condition that is not strictly an error, but still results in no products. This occurs when the user's cleaned event file has no good time. This may happen because various screening parmaeters are extreme and exceed the recommended good ranges.

A cleaned event file with no events and no good time is technically a perfectly valid event file, but it is not possible to produce any pipeline output products. If this condition does occur, the task will print a warning message, and exit with error status 218.

An automated script calling nicerl3-lc can use the exit status to determine a course of action. If the caller detects status code 218, this indicates that the caller should not expect any products and should move on.

Modifications

  • 2022-11-16 - initial draft
  • 2022-12-20 - updated for graphical output issues / options
  • 2023-07-11 - add discussion of normalization