Software Release HEASoft 6.31.1 / NICERDAS 10a and Calibration xti20221001


This page discusses new changes available in NICERDAS version 10, distributed in HEASoft 6.31 (and 6.31.1). This release is a major NICER software and calibration release, with signifigant improvements regarding background subtraction and overall workflow.

Read this thread if you want to: Understand NICERDAS 10 software

Last update: 2022-12-20


Obtaining useful scientific results from NICER observations is a combination of NICER observational data with software and calibration products. Over time, NICER is committed to improving software and calibration so that scientists can get the most out of their data.

This page discusses changes available in NICERDAS 10, released with HEASoft 6.31 on November 15th, 2022, and a subsequent patch release 6.31.1 on December 19th, 2022 (see the 6.31.1 section below for more information on changes in the patch release).

HEASoft 6.31.1 is a major NICER software feature release. There are significant background modeling and workflow improvements. There is also a new NICER calibration release, designated xti20221001, which was also relased in late October. It is recommded to upgrade to the latest patch release (HEASoft 6.31.1 / NICERDAS 10a).

As always, we recommend users to document which version of software they are using, especially for publication. This will help future analysts repeat your results. For more information, see the Reporting NICER software and calibration versions thread.

Overview of NICER Improvements

HEASoft 6.31.1 (NICERDAS 10a) is a major NICER software release, and a corresponding NICER calibration update, xti20221001, is also released. This section provides an overview of the changes. More detail is provided in subsequent sections. The patch release HEASoft 6.31.1 (NICERDAS 10a) has the same featues as HEASoft 6.31, but additional bug fixes.

Standard "Level 3" spectrum and light curve pipeline tools The tasks nicerl3-spect and nicerl3-lc provide a complete and standard spectral and light curve analysis chain. These tools perform all of the NICER-recommended steps to produce spectra (and associated backgrounds and responses) and light curves. The tools can also produce a handy graphical (.png) output. See the Details section below for more information.

Background Modeling. This release includes three background models, integrated into HEASoft. It is no longer required to download background modeling software or auxiliary files separately since they are now included with NICERDAS. See the Details section below for more information.

The background models included are:

  • SCORPEON. This release introduces a new public model named SCORPEON. SCORPEON provides the ability to perform very high quality background modeling in a range of conditions. Unlike other models which produce a background file which is subtracted from the source spectrum, in its more sensitive form SCORPEON produces a background model which can be modeled in XSPEC along with the source parameters.
  • 3C50. The Remillard et al. 3C50 model, as implemented in the nibackgen3c50 tool, is now part of NICERDAS. This tool was previously a separate download and is now integrated into HEASoft and requires no separate downloads.
  • Space Weather. The Gendreau & Corcoran Space Weather background tool is also a part of this NICERDAS release. Like 3C50, the Space Weather model previously required separate downloads and now does not.

Geomagnetic Data Now Required. Geomagnetic quantities are used by two of the three background models. These quantities include KP (interplanetary Kp index) as well as SOLAR_PHI (solar modulation potential). Now using geomagnetic data will be required. Users must download an updated set of geomagnetic data and set their GEOMAG_PATH environment variable to point to the location.

To help users do this process, a new tool called nigeodown is provided which automatically downloads, extracts and installs geomagnetic data. Please see the Geomagnetic Quantities thread.

Because multiple background models use geomagnetic data in support of their background estimates, specifying geomagnetic data is required and no longer optional. Please see the Setting Up a NICER Analysis Environment thread for more information about how to do this practically, as well as the Geomagnetic Quantities thread for additional details. By default, nicerl2 will automatically locate and use the geomagnetic data if it has be set up properly without additional command line settings.

Automatic screening. The nicerl2 task now automatically screens user data for detectors that are noisy or misperforming. This screening is done on a pointing-by-pointing GTI. See the Details section below for more information.

Default overshoot screening. The default overshoot screening criterion have been changed. There are two parameters to nicerl2 (and nimaketime) that control overshoot filtering: overonly_range and overonly_expr, and these were set to aggressively remove any type of background flaring. However, this also had the unwanted effect of sometimes removing all data from an observation, or creating a "shredded GTI" problem for borderline count rates. With background models now being included in NICERDAS, the overshoot screening can be relaxed. The new screening is overonly_range=0-30.0 (instead of 0-1.5, and with no overonly_expr). The nicerl2 task now automatically screens user data for detectors that are noisy or misperforming. This screening is done on a pointing-by-pointing GTI. See the Details section below for more information.

Potential speed enhancements. The nicerl2 task can now be run with incremental=YES, which means that it will only reprocess existing data if necessary. See the Details section below for more information.

Calibration Updates.NICERDAS also is meant to be paired with a new NICER CALDB release, designated xti20221001. In addition to the new CALDB files required to support the above capabilities, this release also has a new energy scale calibration which improves the high energy response (designated "optmv13"). See the Details section below for more information.

Should I Reprocess My Data?

Users will want to know, should they reprocess my data with this new release? The answer depends on the circumstances.

Users new to NICER or starting a new project will likely benefit from reprocessing their data with the newest release. Also, the new background and high level product features in this release are compelling enough that users may want to reprocess their data to take advantage of the features.

Users New to NICER. If you are just starting to use NICER data, the NICER team highly recommends that you begin your work with the newest NICER software and calibration. This release creates much more streamlined workflows for both spectral and light curve extraction, so it is probably not worth investing time learning older techniques.

Starting a New Project. If you are starting a new project, the NICER team recommends that you update your software and calibration. Although the new energy scale released with this calibration update will provide marginal improvements in analysis, the new automatic screening tools may be beneficial to users.

Users who have already developed their own workflow or have already extracted spectra or light curves, may not see a great benefit from going back to reprocess their data. However, the new tools and calibrations in this release will be the basis of future improvements, so moving analysis pipelines to the new release is definitely recommended when practical.

Try Out New Background Models or High Level Product Pipelines. Since three background models are included with NICERDAS, this may provide users an opportunity to try these models out with their data without doing separate downloads. In addition, this release has two new high level ("Level 3") product pipelines, nicerl3-spect and nicerl3-lc, which perform all the recommended steps and make product extraction much simpler. Users may want to try out these tools, which require minimal learning curve and compare to their own workflow.

A separate, but related, question is, am I required to reprocess my data to take advantage of these new features? The short answer is yes, you are required to reprocess your data with nicerl2 to use the new background modeling and high level product extraction tools. This is mostly because certain new filter file columns are required for the new features to work properly, but also because the old filter file generator (NICERDAS 8 and earlier) has some bugs that need repairing.

Potential Incompatibility Gotchas

Generally the NICER team strives to maintain forward and backward compatibility. This improves the user experience with our software and also instills confidence in the results from our observatory. However, in this release, there are the potential for some minor incompatibilities.

Filter File Changes. The filter file (MKF file) columns have changed since the previous release. Some columns have been added and some removed (for more information, please see the Filter Files thread. Although it is advised to reprocess all of your data when starting a new project, there may be some situations where you have to merge filter files from across versions. Use the task nimkfmerge for this situation.

Metadata keywords in NICER spectra. The NICER response generation tasks nicerarf and nicerrmf have the option updatepha. When updatepha=YES, keywords are written to the spectrum file that indicate to XSPEC which background and response files to use (BACKFILE, ANCRFILE, RESPFILE, etc).

Unfortunately, this updatepha=YES feature did not work well in previous NICERDAS releases. Also if you included a path with subdirectories, these subdirectories were also included in the output keywords. These subdirectory names could make it difficult to use the file within XSPEC, especially if your working directory changes between the time that you generate the file and the time that you use the file within XSPEC.

If you did rely upon updatepha=YES working, please beware that subdirectory components are now stripped away when written to the output keywords. You can use the "cd" command within XSPEC to change directories to the location of your data. However, the NICER team recommends to set updatepha=NO everywhere, and explicitly load the files that you need. The nicerl3-spect command will automatically generate a "load" XSPEC script file that will help get you started.

Details: New High Level Spectra and Light Curve Pipelines

NICERDAS 10 includes two new high level product pipeline tasks:

  • nicerl3-spect performs all the NICER-recommended steps to extract a spectrum and generate all required auxiliary files like responses and backgrounds.
  • nicerl3-lc performs all the NICER-recommended steps to extract a light curve.
These tools make producing spectra and light curves much more straightforward than in previous releases. Both tasks perform all the NICER-recommended steps to produce the most useful and accurate products. They are useful for beginners as well as advanced users. Both tools allow the user to choose which of the three supported background model to use for background subtraction.

The new high-level product extraction tools nicerl3-spect and nicerl3-lc provide a straightforward way to obtain accurate spectral and light curve analysis results.

The nicerl3-spect tool performs all of the NICER-recommended steps to extract a spectrum, generate responses (both ARF and RMF), and generate a background using the user's choice of background model. In addition, the task creates a basic XSPEC .xcm file (or pyXspec .py file) which can be used to load the data into XSPEC and diagnose it quickly.

The nicerl3-lc tool extracts light curve for NICER data, and optionally performs background subtraction.

Both tools require that you have reprocessed your data with a current version of nicerl2.

Details: NICER Background Modeling Improvements

HEASoft 6.31.1 / NICERDAS 10a includes major advancements for NICER background modeling.

Three background models are included in NICERDAS 10, including the well-known 3C50 and Space Weather models, and the new SCORPEON model. All three models are integrated within the high-level product tools.

The high level product tools nicerl3-spect and nicerl3-lc allow the user to simply choose which model to use and the tools will automatically generate the products. In addition the user can run the tools separately if they wish.

Existing 3C50 and Space Weather Models Now in NICERDAS. Before NICERDAS 10, two background models (3C50 and Space Weather) were available as a separate download from the NICER Background Estimator Tools page. These models (nibackgen3C50 and nispaceweather) are now included within NICERDAS and do not require separate downloads. All of the software and extra files required by these tools are now part of NICERDAS and NICER's CALDB release. Including these models should make using and trying background models easier for the user.

SCORPEON. A new background model named SCORPEON is also included in this release SCORPEON taks a very different approach to background modeling. You can find more information about this model on the SCORPEON overview page.

In SCORPEON, rather than generating a file which is meant to be subtracted from the spectrum within XSPEC, the SCORPEON approach is to model the background directly without ever having to subtract it. SCORPEON is composed of several components that are likely associated with real physical phenomena in the geomagnetosphere.

The SCORPEON background model is a parameterized background model that can be fit in XPEC along with your source parameters. This has some major benefits. Done properly, you will never have background over- or under-subtraction problems, as you will likely have with other models. In addition, the uncertainties your source spectral parameters will properly reflect the covariance with uncertainties in the background, i.e. a more realistic error estimate. Also, by not subtracting background, you are able to use more appropriate statistics in the low counts regime such as Cash or "pgstat".

If the user insists upon having a background file instead of XSPEC model, SCORPEON can generate this. The user will not get the full benefits of SCORPEON, but in some cases a background file is necessary.

For more information on SCORPEON, please see the SCORPEON overview page.

Existing Background Workflows. Please know that as these tools were integrated into NICERDAS, the existing interfaces for nibackgen3C50 and nibkgestimator were maintained. In other words, you can use the versions released with NICERDAS as drop-in replacements for the versions you downloaded separately. The new NICERDAS versions have some additional capabilities (such as to query CALDB for auxiliary files), but existing command line usages should be unchanged.

Existing background workflows should continue to work, or you can migrate to the high level product pipelines.

Therefore, if you have an workflow that uses the existing 3C50 (nibackgen3C50) or Space Weather (nibkgestimator) tools that you downloaded separately, you have several choices.

  • You can continue to use the separately downloaded versions, but please be aware that these may not be updated in the future.
  • Use the new versions that are integrated in NICERDAS instead of the separately downloaded versions, with little to no usage change.
  • Migrate to use the high level products pipeline tools nicerl3-spect and nicerl3-lc and let those tools extract background for you.

For all background models, the user is required to reprocess their data with a current version of nicerl2.

Details: Automatic Screening

NICERDAS 10 includes a powerful new automatic screening tool named niautoscreen. This tool checks each detector for problematic issues during each pointing good time interval (GTI). Any such detectors are excluded only during the pointings where the problems are not detected, thus giving the benefit of all good detectors when possible.

The new niautoscreen task is run automatically by nicerl2 and will exclude detectors with problematic issues.

The tool adjusts the event file's FPM Selection information. This allows the response generation tools to automatically construct an accurate response, even when some detectors are deselected for a portion of the time.

The automatic screening tool checks various quantities that can indicate possible problems. The tool constructs robust quantities (median, robust standard deviation), which are used to determine which detectors are out-of-family, if any.

The tool checks the following quantities on a per-detector and per-MPU basis:

  • Overshoots. Indication of high non-X-ray background
  • Undershoots. Indication of high optical loading and noise
  • Noise peak rate (0-0.25 keV). Inidication of high noise
  • Soft excess (0.3-0.7). Individual detectors may have shifted noise peak into this soft energy range.
  • Shredded GTI. At high telemetry rates, detector data can be lost, creating thousands of small gaps and GTIs. This is sometimes called "shredded GTIs"
  • "Round Robbin." In some cases, MPUs can slip into "round-robbin" mode where detectors within the MPU are switched on and off every few seconds. This can cause unwanted artificial gain shifts.
In addition, detectors with known problematic issues are screened. These values are stored in a "bad times" table in CALDB.

niautoscreen is automatically run when you run nicerl2. You can control its function by setting autoscreen=YES or autoscreen=NO.

Details: Potential Speed Enhancements

The "Level 2" pipeline task nicerl2 has a new parameter incremental. When incremental=yes, the task will avoid reprocessing data when existing data is up to date.

By setting incremental=YES when running nicerl2, significant time savings can be obtained, especially if the user is re-running nicerl2 multiple times with slightly different screening settings. nicerl2 automatically checks the metadata keywords for the correct information, and if the file is not up to date, reprocessing is forced.

The steps that can be skipped when incremental=YES include re-calibration and filter file generation. Some steps are always done, regardless of the incremental setting.

Please use this setting carefully. If you are upgrading your software or calibration, it is recommended to set incremental=NO to for a complete clean processing the first time, but it can then be set to YES for subsequent runs.

For more information about nicerl2, please see the nicerl2 analysis thread.

Details: Calibration Updates

Along with NICERDAS 10, the NICER team is also releasing a new version of CALDB.

NICER CALDB release xti20221001 contains many new calibration updates required to support the new features of NICERDAS 10. Please see the NICER's Calibration Recommendations for current best practice, and NICER Calibration Documents for calibration release notes.

For example, the required auxiliary files for background generation, and bad times for automatic screening, are included in CALDB.

Also, this release has a new energy scale for both the slow (PI) and fast (PI_FAST) channels. The updates improve the cross-alignment of the energy scale between detectors, especially at high energies above 8 keV. This change has negligible effect below 8 keV (< 5 eV on average). Above 8 keV there are large effects, which increase to a maximum of ~300 eV for some detectors at 13 keV. In practice, few scientists will notice a change, since most NICER science counts are in the 0.3 - 6 keV range.

Other Updates

NICERDAS 10 contains several other significant updates.

Filter file default column names have changed. The new "NICERV4" alias gives the default column names for this release. In general, some low-utility columns have been removed to save execution time and disk space, while some columns have been added. Please see the Filter Files thread for more information.

The NICER team has discovered rare circumstances where an event file may contain events where the PI_FAST column is erroneous. This appears in the PI_FAST spectrum as a group of spectral line-like features around 3 keV. This problem has been traced to events that create an invalid pulse height in the fast chain, and a new tool (nicalfixevt) corrects this problem. This tool is automatically run when nicerl2 (or nicercal) is run.

Complete Change Log

  • New NICER background models included in NICER:
    • 3C50 - nibackgen3C50 now incorporated in NICERDAS
    • Space Weather - nibkgestimator now incorporated in NICERDAS (and HEASoft equivalents niswbkgspect, niswbkglc)
    • SCORPEON - new background modeling tool (niscorpspect, niscorpspectmod)
  • New high level product extraction pipeline tools
    • nicerl3-spect
    • complete NICER spectral extraction pipeline
    • nicerl3-lc - complete NICER light curve extraction pipeline
  • New lower level supporting tasks
    • niextspect - extract standard NICER spectrum
    • niextlc - extract standard NICER light curve
    • niphasyserr - apply standard NICER systematic error vector to spectrum
    • niphaquality - apply NICER QUALITY screeing to spectral channels
    • nihaloem - calculate Halo and Local Hot Bubble emission estimates
  • New tools (nicerl2, niprefilter, niprefilter2) now accept alias NICERV4
  • New tool nicalfixevt fixes invalid PI_FAST pulse heights
  • New task niautoscreen for automatic per-detector screening
  • New task nigeodown
  • All NICER tasks more consistently write HISTORY keywords
  • niexposum - automatically determine which extension and column to operate on
  • nicerarf - only allow response generation on non-barycentered data
  • nivigsum (nicerarf) - now provides a warning when target is out of field of view
  • nicerarf - user can specify R.A. / Dec. as OBJ or NOM to use the metadata keywords embedded in the spectrum file
  • nicerarf / nicerrmf - fix bugs when updatepha=YES, though updatepha=NO is recommended
  • nicerl2 (and niprefilter, nicercal, nimpucal) can reset the RA_OBJ / DEC_OBJ keywords of output products
  • niprefilter2 - now has vercheck parameter which checks if filter file needs reprocessing
  • nicercal (and nicerpi, nicertimecal, nimpucal) now have caldbcheck / incremental keywords to prevent reprocessing if the output product is already up to date
  • niprefilter2 - user can now explicitly request / reject columns with COLNAME or -COLNAME
  • nicerl2 - SOLARPHI column is now SOLAR_PHI
  • nifpmsel - now accepts detlist=@filename.txt syntax to read from file
  • barycorr no longer attempts to barycenter the FPM_SEL extension of cleaned event files, preventing an error
  • nifpmsel - uses more sensible tmin/tmax for empty GTIs, which prevents barycentering failures or lockups
  • nicerrmf, niexposum, niextract-events, nifpmsel - fix off-by-one bug in gtifilter()
  • nicerarf and nicerrmf write ANCRFILE, BACKFILE, RESPFILE with no leading path components (filename only)
  • nimaketime - default overonly_range=0.0-30.0 and overonly_expr=NONE

HEASoft 6.31.1 / NICERDAS 10a Changes

The NICER team highly recommends upgrading to patch release HEASoft 6.31.1 (NICERDAS 10a), since it contains several important bug fixes.

There are major reasons we strongly recommend NICER users install the patch release. These are due to bug reports or incorrect results being presented within XSPEC. The issues that are resolved are:

  • XSPEC residuals plots when using the SCORPEON model were incorrect. This has been fixed.
  • An error that appears " ffgcnn could not find column: SOLAR_PHI" if you have two copies of the filter file (gzipped and non-gzipped) is fixed. More information can be found on our Common Errors page.
  • An error appears when you nicerl3-spect or nicerl3-lc when you try to plot the results using the default PNG output format, and you do not have a PNG graphics driver, now has better reporting. More information can be found on our Common Errors page.

There are other more minor NICER-related changes to HEASoft. The issues noted here are not as critical as the ones above but may affect some scientists.

  • The Space Weather background model tools would report an error if the average number of selected FPMs is non-integer, which is now fixed.
  • A filtering bug for the 3C50 model (nibackgen3C50) when many detectors are disabled has been fixed.
  • nicerl3-spect erroneously reported an error if the UFA file is not found, even if it is not needed for SCORPEON or Space Weather. This bug is fixed.
  • A long standing bug in prefilter leading to the error 'error: satellite has re-entered' for a handful of NICER observations has been fixed. niexposum would report an "undefined subroutine" error, now fixed

Related Topics

For instructions on setting up a NICER analysis environment, please see the Setting Up a NICER Analysis Environment thread.

For more information on recommended settings and calibration recommendations, please see the NICER Calibration Recommendations analysis thread.

For release notes for the previous release (HEASoft 6.30 / NICERDAS 9), please see the Software Release HEASoft 6.30 thread.


  • 2022-10-17 - initial draft
  • 2022-12-14 - fix dangling links
  • 2022-12-20 - updated for HEASoft 6.31.1 / NICERDAS 10a