Complete Light Curve Product Pipeline (nicerl3-lc)Return to: Analysis Threads | Analysis Main Page
OverviewAlmost all users will wish to product a light curve for their X-ray observational data. This thread describes a complete end-to-end spectral product pipeline called nicerl3-lc. Read this thread if you want to: Extract light curve products in a straightforward way Last update: 2024-02-28 IntroductionLight curve analysis is key to most X-ray analysis. In some cases it provides context for analysis. For example, if there are bursts or eclipses, the scientist can use an X-ray light curve to detect these occurrences. In other cases, light curve analysis is the beginning of timing analysis, such as power spectroscopy. Thus, extracting a light curve is a fundamental task that almost all NICER users will need to do. This thread describes how to extract all NICER light curve products using a single pipeline task called nicerl3-lc. This task generates all products needed for light curve analysis: light curve and (optional) background estimate. The nicerl3-lc task became available with HEASoft 6.31, which was released in November, 2022. The task provides a straightforward and streamlined way to extract products with NICER-recommended methods. While nicerl3-lc is targeted at the "new user" of NICER data, it is equally usable by advanced users. Many of the complicated and/or poorly documented steps are now done automatically.  Figure 1. Workflow showing nicerl3-spect and nicerl3-lc to make standard spectral and light curve products. Figure 1 shows the basic workflow for any NICER analysis using the new nicerl3 high level pipeline tasks. The user first runs nicerl2 to apply standard calibration and screening to the data, and then runs the nicerl3-lc task to create light curve products. For more information about the "levels" of NICER analysis, please see the NICER Processing Levels thread. The input of nicerl3-lc is the output of nicerl2. Namely, the clean event file ("cl" file), plus other auxiliary files like the filter file (MKF file). Some background models, such as 3C50, also use the unfiltered event files ("ufa" files). Without special prompting, the nicerl3-lc task automatically recognizes the directory layout of a NICER standard observation as well the situation where the user directed nicerl2's outputs to a different directory. The nicerl3-lc task will make many automatic decisions for you, such as choice of background model, but the advanced user is free to make alternate choices, as described below. As output, nicerl3-lc makes the following products:
nicerl3-lc also creates a graphical plot of the light curve. This is suitable for quicklook pipelines that want a standard view of the light curve. nicerl3-lc chains together multiple standard NICER tasks in the team-recommended fashion. Although not recommended, users can also run the tasks manually themselves as well. The tasks are,
Although individual manual tasks are provided, the NICER team recommends to use nicerl3-lc to generate all products in a consistent manner. The following discussion shows the basic operation of nicerl3-lc. For more detailed information on task operation, please see the nicerl3-lc help file PrerequisitesThe nicerl3-lc task requires the output of the nicerl2 task. This means EITHER, the data processed "in-place" in an existing observation directory, or an output directory as described in the "WORKING WITH READ-ONLY INPUT DATA" section of the nicerl2 help file. Basically, the task will look in the supplied input directory name for the subdirectory xti/event_cl, or in the subdirectory itself, for the required files. The required files are described above. Extract Light Curve ProductsRun the nicerl3-lc spectral product pipeline task with the following command.
nicerl3-lc 1234567890 pirange=300-1500 timebin=60.0 clobber=YES
where
Upon completion, nicerl3-lc will product the products as described
above. They will be placed in the output directory. As an example, let's list the
directory and see what products were produced by this run.
As you can see, a light curve is produced. By default no background is produced unless explicitly requested. Plot Light Curve with 'fplot'
We can make a quick plot of the light curve using the fplot command.
Here we are using TIME as the X-axis, RATE as the Y-axis, and the /xs
"X windows" plotting device. We use "offset=YES" so that the X-axis
is numerically readable. The result is shown in Figure 2.
Of course, in this example the observation ID number is 1234567890;
this will vary according to your particular observation. If you are
using an alternate output directory, you may need to
use Light Curve NormalizationAs of NICERDAS 11, the nicerl3-lc task will automatically normalize light curves by the number of enabled and/or selected detectors. This has the added benefit of having NICER standard light curves be more representative of the source's true flux. If the number of enabled and/or selected detectors varies during an observation those would be visible in the raw light curve as jumps, which are indicative of variations in the number of detectors rather than the source's flux. NOTE: the renormalization capability described in this section was not performed in HEASoft releases before July 2023 (NICERDAS 10 / HEASoft 6.31.1 or earlier). For those earlier releases, the light curves were not normalized, and they contain the raw array count rate, equivalent to detnormtype=none. How to enable. Renormalization is enabled by default in NICERDAS 11 / HEASoft 6.32 or later. Disable normalization to recover earlier behavior by setting detnormtype=none. How light curves are normalized. By default, light curves are normalized to the count rate of an equivalent 52-detector array. For example, if 26 detectors are selected in an observation, the raw count rates are multipled by a factor of 2. The filter file and/or the FPM Selection information accompanying the data are used to determine the normalization on a second-by-second basis. Thus, observations where the number of enabled detectors varies from time to time are accomodated. The parameter detnormtype determines the normalization factor. Set detnormtype=arr52 to normalize to an equivalent 52-detector array. Set detnormtype=fpm to normalize to an average enabled detector. When not to normalize. There are some science cases where normalization may not be the best approach to light curve analysis. One may be to maintain compatibility to a previous analysis when this capability was not available. Another may be for sensitive timing analysis where raw counts are preferred. In these cases, set detnormtype=none. Edge effects. NICER detector housekeeping is not 100% accurate. Detectors are not enabled immediately when commanded to start taking data: there may be some brief transient behavior. Also, detector housekeeping, which is reported once per second does not capture where in that second the on/off transition occurred. Therefore, detectors that transition between on-off states are "blanked out" for a period around the transition time. The amount of blanking is governed by the detnormbuff parameter (whose default is 1.6 seconds). Set detnormbuff=0.0 to use all time bins, including potential temporal transients during detector start-up. Also, small changes in numbers of enabled detectors do not have this blanking. This behavior is governed by the detnormchg parameter, whose default 10 in units of 10%. In other words, for a 52-detector array, changes by 10% (5 detectors or fewer) wlil be fully included with no blanking. Set detnormchg=0.0 to blank out all detector changes. The default normalization as of NICERDAS 11 / HEASoft 6.32 is to scale to a 52-detector array count rate. Use detnormtype=fpm to scale to per-detector rate. Disable normalization by setting detnormtype=none (which is equivalent to what happens in earlier releases). Variation: Applying Additional Screening to DataA common situation will be that the user desires to apply additional temporal screening to their data. The cause of this is typically scientifically-motivated. For example, a user might want to look at the spectrum of a burst, or during eclipse, or during a high or low state. Generally speaking the user will have to construct a GTI (Good Time Interval) file which contains the desired time range. You may be able to use maketime or nimaketime to get a new GTI file, or you may have to construct a GTI file manually yourself. For an example of how to do this (for Swift) please see the Swift BAT thread Making BAT Light Curves (but please beware that for NICER MJDREFI=56658 and MJDREFF=7.775925925925930E-04). Once you have a GTI file, you can apply it simply from the nicerl3-lc
command line,
If you want to apply different quality screening levels, such as overshoots or undershoots, it is recommended that users run nicerl2 again. To reduce execution time, you can set the nicerl2 parameter tasks=SCREEN and it will only perform the screening step again, without performing the other more time consuming steps. Variation: Choosing Different Background ModelsTHe NICERDAS suite contains multiple NICER background models. You can choose to enable background models when you run nicerl3-lc, but there are limitations as described below. Beginning with HEASoft 6.33 / NICERDAS 12, NICER supports light curve estimtes from both the SCORPEON and Space Weather background models. Background estimation for light curves is disabled by default: you will have to choose a background model explicitly using the bkgmodeltype parameter to enable background estimation. As of the HEASoft 6.33 release, both the SCORPEON and Space Weather background models are supported for light curve estimates. Please note that by default, no background estimates are generated. Scientists will need to explicitly enable background estimation by using the bkgmodeltype parameter. A the default setting, bkgmodeltype=NONE, no background estimated are retrieved. The bkgfile is not created, and no background estimates are attached to the resulting light curve file. Space Weather The Space Weather model due to Gendreau and Corcoran is distributed with NICERDAS since HEASoft 6.31. All software and calibration files are included within the NICER environment, with no separate downloads required. Settings "bkgmodeltype=sw" (for more details see nispaceweather).
For example, to run with the Space Weather model, use the following command,
The Space Weather model uses a library of background events, and is trained upon cutoff rigidity (COR_SAX) and Kp (KP_POTSDAM). The model filters the library for each light curve time bin, and thus may incur a significant time and computing penalty for light curves with many bins. The Space Weather model does not adjust for threshold changes that occur during orbit day after the May 2023 optical light leak event. In the Space Weather output bkgfile, the RATE and ERROR columns represent the estimated background rate and statistical error. The statistical error is based upon the counts in the library of background events. SCORPEON. (bkgmodeltype=scorpeon) This model been available since HEASoft 6.33 and is due to Markwardt.
For example, to run with the SCORPEON model,
first run nicerl3-spect once; the light curve tool
requires response matrices generated by the spectral product tool.
Then run the following command.
The SCORPEON model uses the same model as is used for spectroscopy, but the model is not "fittable" in the same way. The SCORPEON model predicts normalizations for each SCORPEON component (COR, SAA, etc) at each 1-second housekeeping sample. It also predicts the contribution of each component's normalized spectral shape in the requested light curve energy band. These two contributions, the spectral shape and time varying normalization, are multiplied to retrieve an estimated background contribution in the requested energy band of each background component. For light curve bins larger than 1.5 seconds, the 1-second background estimates are rebinned to match the light curve. For light curve bins shorter than 1.5 seconds, interpolation is used. Thus, performance and quality are not compromised at any light curve bin size. The SCORPEON model can adjust for threshold setting changes that are implemented by the NICER team after the May 2023 light leak event. However, the model is only able to accomodate one setting at a time, either day-only or night-only. Thus, for data observed after the May 2023 light leak event, users are advised to extract separate products for orbit day and orbit night portions. See the nicerl2 page for more information how to achieve this. To estimate backgrounds, the nicerl3-lc tool requires response information for the diffuse sky, as well as detector response matrices for X-rays and backgrounds. Rather than recompute these files, the tool accepts them as input. For this reason, you must run nicerl3-spect before running nicerl3-lc. If you use the default output file names for nicerl3-spect, then nicerl3-lc will automatically find and use them. If you change the names of these responses to non-default values, then you will have to use the same parameters (skyarffile, rmffile, bkgrmffile) to name them explicitly. Before running nicerl3-lc, you must run nicerl3-spect to generate response files. In the SCORPEON output bkgfile, the BACKV and BACKE columns represent the estimated total background rate and systematic error. The systematic error is set to 50% by default. In addition, the per-component background estimates are provided in separate columns with names such as BACK_COR, BACK_SAA, and so on. Constant background rate estimates are reported on the BACK_BASELINE column. In this way, users can diagnose which components of the background may be most strongly affecting their data. Please be aware that the SCORPEON model quality is not high enough to do a simple subtraction from light curve samples. Instead, it is recommended to plot the background rate estimates (BACKV,BACKE) alongside the light curve rate samples so that scientists can visually inspect the two together. When background estimates are requested, they will be reported in the separate bkgfile. In addition, the light curve itself will contain an additional column called BACKV for the background estimate (and BACKE for the estimated background error, if any). Each of the background models has additional parameters or settings that allow more in-depth control of the model generation. nicerl3-lc does support these. For the SCORPEON model, the bkgcomponents, bkgvariant, bkgsoftlanding and bkgver parameters are passed to the SCORPEON modeling tasks, which are documented on the SCORPEON help file pages. For the 3C50 model, the hbgcut and s0cut parameters that control screening of extreme conditions are passed to the nibackgen3C50 model, which are documented on the nibackgen3C50 page. For the Space Weather model, there are no additional parameters. Variation: Changing Energy Range or Time Bin SizeUsers will naturally want to choose the time binning and energy range that is right for their scientific project. The energy range is given by the "pirange" parameter, which has no default. The user must specify an energy range as a PI channel range "start-stop". Each NICER PI bin is 10 eV (see Gain Calibration for more details), so a PI range of 300-1500 is equivalent to 3.0-15.0 keV. The time bin size must be given in units of seconds.
As an example, if the scientist wishes a 0.3-2.0 keV light curve with
100 second bins, they could use the following command:
Variation: Choosing Different Graphical Output OptionsBy default, the nicerl3-lc task produces a PNG output file. The user can make other plotting choices. To completely disable an output plot, use the doplot=NO option.
To choose different graphical output formats, use the
plotfiles parameter. For example, to output both postscript
and PNG files, use the following command line.
Please note that the default graphical output format is PNG. If your computer does not have a PNG graphical library installed (typically "libpng"), you may experience an error. If you can, you may wish to install a PNG output library or driver for your computer, or you may wish to try an olternative graphical output as described above. By default the X-axis is TIME. However, in some cases, it may be difficult to view the details of such a light curve because NICER's observations are often temporarlly disjoint. To plot versus sample number instead of time, set plottime=sandwich, and then all samples will be sandwiched together. Variation: Choosing Different File Namesnicerl3-lc will choose sensible file names for its output files, but you can modify this.
By default, nicerl3-lc will notice the name of the cleaned event
file and use that file name and subdirectory as a basis for all output
file names. For example, if the cleaned event file name is
If you are trying different variations with the same basic inputs,
consider setting the suffix parameter to avoid overwriting existing
files. For example, if you are comparing different energy ranges
with the same inputs, you could run nicerl3-lc twice with two
different suffix values. In this case the following commands would
achieve this,
If you want to take complete control of the process, you can set the output file name options explicitly to what you desire. Please see the nicerl3-lc help file for more information. What Can Go Wrong: No Good TimeGenerally speaking, nicerl3-lc can terminate due to an error condition. Please read the error message for more information about how to remedy the situation. However, there one special condition that is not strictly an error, but still results in no products. This occurs when the user's cleaned event file has no good time. This may happen because various screening parmaeters are extreme and exceed the recommended good ranges. A cleaned event file with no events and no good time is technically a perfectly valid event file, but it is not possible to produce any pipeline output products. If this condition does occur, the task will print a warning message, and exit with error status 218. An automated script calling nicerl3-lc can use the exit status to determine a course of action. If the caller detects status code 218, this indicates that the caller should not expect any products and should move on. Modifications
|
- HEASARC Director: Dr. Andy Ptak
- NASA Official: Dr. Andy Ptak
- Web Curator: J.D. Myers
- Last Modified: 28-Feb-2024