Search:

[Advanced Search]



Software Release HEASoft 6.33.2 / NICERDAS 12a / xti20240206

Overview

This page discusses new changes available in NICERDAS version 12a, distributed in HEASoft 6.33.2, which is a bug fix to HEASoft 6.33. This release is an intermediate NICER software and calibration release, with signifigant improvements to light curve background modeling and spectral modeling of orbit-day data after the optical light leak. This release is intended to be used with NICER calibration xti20240206, which is released in coordination with HEASoft 6.33.

Read this thread if you want to: Understand NICERDAS 12a software

Last update: 2024-04-25

Introduction

Obtaining useful scientific results from NICER observations is a combination of NICER observational data with software and calibration products. Over time, NICER is committed to improving software and calibration so that scientists can get the most out of their data.

This page discusses changes available in NICERDAS 12a, released with HEASoft 6.33.2 in April 2024. Most of this document applies to the original HEASoft 6.33 (NICERDAS 12) release, which was distributed in February 2024.

IMPORTANT NOTE: The patch release HEASoft 6.33.2 contains a bug fix for the nicerl2 task. Upgrading is recommended.

Compared to HEASoft 6.32, this release is an intermediate NICER software feature release. The biggest improvements are to response and background estimates for data taken after the optical light leak episode of May 2023. There are other beneficial improvements that are worth checking out, as detailed in this article. In addition, a new NICER calibration release, with CALDB designator xti20240206 is required to use the new funcitonality.

As always, we recommend users to document which version of software they are using, especially for publication. This will help future analysts repeat your results. For more information, see the Reporting NICER software and calibration versions thread.

Bug Fix Patch Release HEASoft 6.33.2

In April 2024, HEASARC released HEASoft 6.33.2 (NICERDAS 12a), which supercedes HEASoft 6.33 (NICERDAS 12). This patch release contains a NICER bug fix to the task nicerl2.

HEASoft 6.33.2 (containing NICERDAS 12a) is a "patch" release that fixes a NICER bug, among other things. This is a software-only bug fix; no new NICER calibration files are released.

The bug relates to the standard pipeline processing task nicerl2: the task silently ignores the 'gtifiles' parameter and processes the entire data set instead of the user-requested GTI. After upgrading to the patched version, nicerl2 properly uses the gtifiles parameter.

The "gtifiles" option allows the user to supply their own Good Time Intervals (GTIs), which is useful for extra screening, or when using the 3C50 background model on a subset of your NICER data, since 3C50 can't accept a separate GTI file.

If you never intend to use the gtifiles parameter of nicerl2, then you do not need to upgrade. However, since this is a "silent" bug, i.e. producing the wrong results without a warning or error, the NICER team does recommend that you upgrade your HEASoft system. You have two choices to upgrade, an easy "patch installer" or a complete new installation. Either choice will provide the nicerl2 bug fix. If you just wish to upgrade NICER tools, there is a "patch installer" available from this patch release page. The patch installer does not require you to rebuild your entire system or re-build from source code.

Other changes to HEAsoft 6.33.2:

  • compatible with more strict compilers such as gcc version 14 (distributed with Fedora 40)
  • upgrade to more recent Xspec patch level 12.14.00h
These changes are NOT included with the patch installer. If you need these updates, you will need to download a complete new release. For setup instructions, please see the Setting Up a NICER Analysis Environment page.

Overview of NICER Improvements

HEASoft 6.33.2 (NICERDAS 12a) is an intermediate NICER software release. This section provides an overview of the changes. More detail is provided in subsequent sections.

NICER response and SCORPEON background estimates for data taken during orbit day after the optical light leak of May 2023. As detailed on the optical light leak pages the NICER team has responded to the optical light leak by setting the low energy threshold to a higher value (~0.38 keV) during orbit day. The NICER response tool (nicerrmf) and SCORPEON background tool (niscorpspect / nicerl3-spect) now provide estimates taking these changes into account. See the section below for more important details.

NICER SCORPEON background estimates for light curves. A much-requested feature is to have SCORPEON background estimates for light curves. This functionality is now present, using the standard nicerl3-lc tool (and niscorplc). The default is still to not estimate any background, but SCORPEON estimates can be enabled simply. Please see the section below for more details.

The 3C50 background model has been updated for the "2022" gain update. This update covers data with the "optmv13" gain calibration which was released in 2022, as supplied by Ron Remillard and Michael Loewenstein. Previously users with optmv13 gain calibration were directed to the 3C50 background estimates from the optmv12 solution. In practice this gain solution should provide minimally different background estimates. However, in addition to a new gain solution, this new library uses enhanced filtering to remove noisy detectors and noise ringers to improve the fidelity of library backgrounds.

Better estimates of some kinds of flaring. This release has a tool named niprelflarecalc, which can be used to help diagnose flaring. In future releases the estimators produced by this task may be placed in the filter file and/or SCORPEON background estimtes but for the time being, it is a standalone tool. A longer discussion of these new tools is found on the NICER Flares page.

Calibration Updates? This release is intended to be used with a companion calibration release, xti20240206. This CALDB release is available from HEASARC, and users can refer to the Setting Up a NICER Analysis Environment page for more details on updating.

To take advantage of the new software features, users must update HEASoft to version 6.33.2 and CALDB to xti20240206.

If users update HEASoft but not CALDB, or CALDB and not HEASoft, the software will continue to work as before, but new functions will not be available.

Should I Reprocess My Data?

Users will want to know, should they reprocess my data with this new release? The answer depends on the circumstances.

Users new to NICER or starting a new project will likely benefit from reprocessing their data with the newest release. In addition, users working with data taken after the May 2023 optical light leak, or working with data with very high undershoot levels, will obtain significant benefits from reprocessing with this release. Users wishing to analyze data taken during orbit day, and those wishing basic background estimates for light curves will benefit especially.

Re-running nicerl2 From Scratch Not Necessary. Since this release does not contain changes to the filter file or event calibration, you do not need to perform these time consuming steps again. If you have processed your data with HEASoft 6.32.1 (NICERDAS 11), the processed results are still usable. Some of the screening criteria have changed in HEASoft 6.33.2. Therefore, we still recommend re-screening your data using nicerl2 ... tasks=SCREEN but that process is typically very fast compared to the other steps that nicerl2 performs. For more information about the "tasks" parameter, please see the nicerl2 page.

Users New to NICER. If you are just starting to use NICER data, the NICER team highly recommends that you begin your work with the newest NICER software and calibration. This release creates much more streamlined workflows for both spectral and light curve extraction, so it is probably not worth investing time learning older techniques.

Starting a New Project. If you are starting a new project, the NICER team recommends that you update your software and calibration.

Working with High Optical Light Loading. NICER experienced an optical light leak on May 22, 2023, which significantly increased the chances of high optical light loading during orbit day, depending on sun angle. The potential for optical light loading has always existed since NICER launched, but the likelihood to extreme conditions has increased significantly after the light leak developed. This release has several capabilities that aid in the process of screening, interpreting and analyzing data taken those circumstances. It is highly recommended to reprocess data observed after May, 2023, or under high optical loading conditions.

A separate, but related, question is, am I required to reprocess my data to take advantage of these new features? The short answer is no, because there are no changes to the filter file or event calibration in this release. The NICER team does recommend running nicerl2 with new screening criteria as described above.

Potential Incompatibility Gotchas

Generally the NICER team strives to maintain forward and backward compatibility. This improves the user experience with our software and also instills confidence in the results from our observatory. However, in this release, there are the potential for some minor incompatibilities.

Filter File Changes. There are no changes to the filter file columns in this NICER release.

SCORPEON's Default Background Version is v23. This new version exposes new background parameters in the XSPEC model that were not available in v22, which was the previous public release. Although the underlying background estimates are the same between v22 and v23, the two cannot be used interchangeably because they have different XSPEC parameters. Users can continue to use background model v22 by setting bkgver=v22 when running nicerl3-spect, but they won't get the new functionality in this release such as modeling of low energy threshold. For more discussion about how SCORPEON handles versioning, please see the SCORPEON Versions and Variants page.

The 3C50 background model applies to all data calibrated with the "2022" (optmv13) energy scale calibration. Previous releases selected the 3C50 model applicable to the optmv12 gain calibration for any data calibrated with optmv12 or optmv13. If users reprocess existing data, they will find differences in 3C50 background estimates, because a new 3C50 model applies. Users can recover the old (lower-fidelity) backgrounds by running nicerl3-spect with "gainepoch3c50=2021".

New screening by default. Users should be aware that several screening options are enabled by default.

  • max_lowmem=0 - this nicerl2 / nimaketime max_lowmem screening, introduced in HEASoft 6.32, is now disabled by default. Instead the NICER team recommends the new default niautoscreen filtering, which will only disable poorly-behaving individual MPUs, rather than all data.
  • lowmemscr - this new nicerl2 / niautoscreen screening criterium takes the place of max_lowmem. It screens individual MPUs for LOWMEM FIFO errors (lost events) with a default maximum of 200 events per pointing GTI.
  • threshfilter=NIGHT - this new nicerl2 / nimaketime screening, keeps only data taken with the threshold setting of orbit night. Before the optical light leak of May 2023, the "NIGHT" threshold setting was used for all parts of the orbit. Use "threshfilter=DAY" to select a day-only spectrum. It is not recommended to mix threshold settings with threshfilter=ALL. The old filter, thresh_range, can still be used, but is not recommended.
  • thresh_range=FILTER - this nicerl2 / nimaketime screening criterium, introduced in HEASoft 6.32, can still be used, but the NICER team now recommends to use the threshfilter parameter instead.

Details: Dealing with Threshold Changes

As of May 2023, NICER has experienced damage to its optical blocking films which has led to increased levels of optical loading on its detectors. To mitigate this, the NICER team has taken several steps to reduce the impacts. One of these changes it to alter the low energy threshold during NICER's orbit day period. For more information on this topic please see the optical light leak pages.

In short, NICER's threshold is the detector lower cutoff energy. X-ray photons below this energy are not detected (although we discuss complications to this statement shortly). Thus, changes to the threshold change the effective NICER energy range.

The impact of changing NICER's low energy threshold during orbit day is profound. At the new orbit day threshold setting of +35 ADUs (meaning, all thresholds have been increased by 35 data units), the effected threshold has been raised from about 0.248 keV to about 0.38 keV. This change means that while NICER was previously sensitive to X-rays in the range of 0.25-15 keV during all portions of its orbit, it is now sensitive to a reduced range of 0.38-15 keV range during orbit day. While the reduction may not appear large when displayed like this, for some science, which relies heavily upon the 0.25-0.38 keV range, this loss is measurable.

After the optical light leak onset, the NICER low energy threshold is adjusted upward every orbit day. This will force scientists to treat NICER data taken during orbit day and night separately.

The above statements are somewhat simplified. In reality, the threshold is not a sharp function. Instead, it is a smooth function, modeled as a cumulative Gaussian, or error function. The centroid and width of this function change with both optical loading and threshold setting. NICER response and background tools must model this function's behvaior properly in order to provide accurate estimates.

Before this release, the NICER tools did not provide any tools or estimates capable of dealing with threshold changes, other than shifts due to optical loading. This new software release, coupled with the new CALDB release, is able to adjust for threshold settings.

With this new release, nicerrmf is capable of adjusting its low energy threshold to accomodate threshold setting changes. It is capable of estimating a threshold from any mix of threshold settings. This capability is included when running nicerrmm or nicerl3-spect automatically, without intervection by the user. This functionality requires updating CALDB to xti20240206.

Also with this release, the SCORPEON model is capable of estimating backgrounds that have the proper low energy threshold. However, this model is limited to one setting at a time, either orbit day or orbit night. I.e. multiple settings can't be combined.

The 3C50 and Space Weather background models have no ability to accomodate low energy threshold changes. Background estimates from these tools will appropriate for data taken after optical light leak during orbit night only. And users will need to take special care to manually ignore data in the 0.25 - 0.5 keV range if they include orbit day data after the optical light leak.

To better help scientific users deal with these issues, nicerl2 and nimaketime now have the parameter threshfilter, which can be set to "DAY", "NIGHT" or "ALL". For users of the 3C50 or Space Weather background model, we recommend setting threshfilter=NIGHT. For users of the SCORPEON background model, we recommend analyzing orbit-day and orbit-night portions separately by running nicerl2 twice, once with thresfilter=NIGHT and a second time with threshfilter=DAY.

HEASoft 6.33.2 / NICERDAS 12a provide new tools to estimate more accurate responses and backgrounds during data taken after the optical light leak of May 2023 during both orbit day and night. The NICER responses and the SCORPEON background model are capable of adjusting to threshold changes. The 3C50 and Space Weather background produce "night-only" background estimates.

The following pages discuss more about the optical light leak and how to deal with it.

Details: SCORPEON Background Estimates for Light Curves

Before this release, scientists could obtain SCORPEON background estimates for spectra, but not light curves. In this release, SCORPEON background estimates can now be produced for light curves as well.

Before this release, the Space Weather background model was able to produce background estimates for light curves. However, given its internal mechanisms, it was slow for light curves with many bins, and tended to produce noisy estimates when bins are small.

The new SCORPEON light curve background estimates can be produced quickly, and with little degradation or slowdown when bin sizes become smaller.

The background estimation tool is called niscorplc, and nicerl3-lc is capable of using this tool to attach background estimates to your light curve.

IMPORTANT NOTE: nicerl3-lc's default is to not estimate any background for light curves. When running this tool, set bkgmodeltype=SCORPEON in order to enable SCORPEON background estimates.

While the NICER team is releasing this tool to the public, please beware that background estimation is in imperfect art and it is unlikely that any background tool, including SCORPEON, will provide a background estimate that can be simply subtracted from the data. By default, the systematic uncertainty attached to background estimates is 50%.

Background estimates are never directly subtracted from the data by NICER's standard tools. Users should use the background estimates as a way to manually screen the data and to help diagnose flaring or other strange behavior.

The background estimate is placed in the BACKV column and its sytematic error is placed in the BACKE column. As noted above, the default systematic error is 50% (of the background amount).

In addition to BACKV and BACKE, the standard tools will also place columns named BACK_X where X is the name of each SCORPEON background component. These individual components can be used to further assign "blame" to a particular background component when screening light curves.

The background tool natively estimates backgrounds with 1 second time bins. For user time bins larger than 1.5 seconds, the estimator tool automatically rebins the bins to match your light curve. For if your light curve has shorter time bins, then the tool will interpolate the 1 second samples to match your light curve.

Details: Calibration Updates

Along with NICERDAS 12, the NICER team is also releasing a new version of CALDB, designated xti20240206..

NICER CALDB release xti20240206 contains several calibration updates required to support the new features of NICERDAS 12. Please see the NICER's Calibration Recommendations for current best practice, and NICER Calibration Documents for calibration release notes.

In this release, the calibration detector parameter file has been updated to include information about how the low energy threshold behaves as a function of threshold setting (the setting is reported in ADU units). This information is used by nicerrmf to provide an improved trigger efficiency curve for data observed after the optical light leak of May 2023 during orbit day. The other parameters in this file have not changed, so users should not expect other changes to resopnses, or for data taken during orbit night, compared to previous releases.

Also, in this release contains the 3C50 background model corresponding to the "optmv13" energy scale calibration (which 3C50 calls the "2022" gain scale). When reprocessing data, the software will automatically pick up this new calibration.

As a minor note, two X-ray transmission curves stored in CALDB were found to be mis-labeled, and this has been corrected. Since both curves are multiplied by each other during ARF generation, the mis-labeling had little practical or numerical importance.

Complete Change Log

  • nicerrmf now calculates variable threshold based on in-flight setting
  • nicerl3-lc now can estimate SCORPEON light curve backgrounds with niscorplc, but the default is still no background estimate
  • nicerl2 / nimaketime have new parameter threshfilter which allow users to select day-only or night-only thresholds, useful after the optical light leak of May 2023
  • nibackgen3C50 adds new gainepoch (2022) for that year's gain solution
  • NICER SCORPEON model adds background version v23 (bkgver=v23) which includes a low energy threshold parameter (nxb), as well as parameters of the detailed nxb variant to better fit the high energy shape of the prel component.
  • niautoscreen has a new parameter lowmemscr to do per-MPU LOWMEM screening
  • nimaketime disables max_lowmem by default, in favor of niautoscreen
  • nicerl3-spect and niphasyserr have new parameter syserrfactor so user can scale systematic error by desired factor
  • niprelflarecalc (NEW) can estimate precipitating electron ("PREL") flares.
  • nigti (NEW) allows uers to make GTI from text file or command line
  • niscorplc (NEW) which computes background estimates for light curves
  • small changes and bug fixes:
    • SCORPEON cosmic_ray calculation handles NaN values correctly
    • niscorpspectmod uses correct background response file name
    • nicerrmf updates DETCHANS keywords in both response extensions
    • niextract-events bug fix to handle case of entire MPU off/disable
    • internal library fix to expand_item_list prevents niextract-events infinite loop
    • nicerl3-lc avoids emitting extraneous file named "1"
    • internal tool niscorpcalc now emits TIMEDEL/TIMEPIXR keywords
    • niscorpcalc correctly computes noise peak centroid when noise_enom != 0 (which is rare)
    • nicertimecal has slight verbosity changes
    • nimpucal now uses timebiascalfile=CALDB instead of NONE, but most users will not see this since nimpucal enforced that behavior anyway
    • nicerl3-lc, nicerpi, nicerrmf, nicertimecal use internal NicerTime libraries for consistent behavior
    • nicerl3-spect adds gainepoch3c50 parameter so user can manually set gain epoch if desired
    • nimaketime bug fix to restore outexprfile functionality lost in NICERDAS v011

Related Topics

For instructions on setting up a NICER analysis environment, please see the Setting Up a NICER Analysis Environment thread.

For more information on recommended settings and calibration recommendations, please see the NICER Calibration Recommendations analysis thread.

For release notes for the previous release (HEASoft 6.32 / NICERDAS 11), please see the Software Release HEASoft 6.32 thread.

Modifications

  • 2024-02-13 - initial draft
  • 2024-04-25 - updated for HEASoft 6.33.2 (NICERDAS 12a)