Software Release HEASoft 6.32 / NICERDAS 11


This page discusses new changes available in NICERDAS version 11a, distributed in HEASoft 6.32.1. This release is an intermediate NICER software and calibration release, with signifigant improvements. Users will find many of the changes beneficial, especially for data taken after May 2023.

Please note that HEASoft 6.32 / NICERDAS contains a bug that results in incorrect normalization and screening of data. Please upgrade all installations to HEASoft 6.32.1, which was released on August 23, 2023.

Read this thread if you want to: Understand NICERDAS 11 software

Last update: 2023-08-23


Obtaining useful scientific results from NICER observations is a combination of NICER observational data with software and calibration products. Over time, NICER is committed to improving software and calibration so that scientists can get the most out of their data.

This page discusses changes available in NICERDAS 11a, released with HEASoft 6.32.1 in August 23, 2023. A previous release of July 2023, HEASoft 6.32 / NICERDAS 11, contains a bug.

HEASoft 6.32.1 is an intermediate NICER software feature release. While no major calibration or workflow changes are provided, there are significant accomodations for data taken after May 2023, when NICER developed an optical light leak. There are other beneficial improvements that are worth checking out, as detailed in this article.

Any users of HEASoft 6.32 should upgrade to HEASoft 6.32.1 immediately due to a bug in the previous version which results incorrect normalization.

As always, we recommend users to document which version of software they are using, especially for publication. This will help future analysts repeat your results. For more information, see the Reporting NICER software and calibration versions thread.

HEASoft 6.32.1 Patch Release Fixes Incorrect Normalization

The original HEASoft 6.32 release contains a bug which results in incorrect normalization in many cases. The bug occurs in the task 'nifpmsel', and impacts data where per-MPU screening is requested by the user or by automatic screening. The bug affects only HEASoft 6.32, and not earlier or later released versions.

More information about the bug can be found on the MPU Autoscreening Bug page.

The only remedy is to upgrade to HEASoft 6.32.1, which contains NICERDAS 11a. NICERDAS 11a / HEASoft 6.32.1 is available from the HEASoft download page. If you need help installing your NICER analysis environment, please see the NICER Setup page.

Overview of NICER Improvements

HEASoft 6.32 / 6.32.1 (NICERDAS 11 / 11a) is an intermediate NICER software release. This section provides an overview of the changes. More detail is provided in subsequent sections.

Standard "Level 3" light curve tools now correct for number of selected / enabled detectors. The task nicerl3-lc has been improved to renormalize a light curve based on the number of selected detectors. This handles the case when users are examining a data set where the number of detectors may change from time to time. This allows light curves to more accurately reflect the flux variations of the X-ray source, rather than variations due to detector turn on-offs. See the nicerl3-lc page for more details.

New screening for high optical loading (undershoot) conditions. The nicerl2 task and its subordinate tasks have been improved to deal with conditions that may be seen under very high optical lighting conditions. The potential exists for such situations to occur at any time during NICER's history, but optical light loading is much more prevalent after May 2023, when an optical light leak developed. The filter file has been augmented with new columns (NICERV5) in order to aid this screening. More details can be found below in the "Dealing with High Optical Loading" subsection.

New way to combine observation data with the niobsmerge task. Merging observation data has been a commonly-requested functionality which is included in this release. The task niobsmerge combines filter files, event files (both UFA and CL files), as well as the orbit file if available. This allows user to more flexibly work with data that spans many observation segments (many days). More detail on this new feature can be found on our Combining Multiple Observations page.

Avoiding "shredded" GTIs. NICER event data can ocassionally have many small GTIs, known as "GTI shredding." These are symptoms of either extreme count rates, or screening on a noisy parameter. This release employs several new techniques to alleviate the problems of shredded GTIs. For more information on this, please see the Preventing Shredded GTIs subsection below.

Calibration Updates?This release does not have a companion calibration release. The NICER team continues to work on calibration issues, especially related to the optical light leak that developed in May 2023, but they are not mature enough to distribute at this time.

Should I Reprocess My Data?

Users will want to know, should they reprocess my data with this new release? The answer depends on the circumstances.

Users new to NICER or starting a new project will likely benefit from reprocessing their data with the newest release. In addition, users working with data taken after the May 2023 optical light leak, or working with data with very high undershoot levels, will obtain significant benefits from reprocessing with this release.

Users New to NICER. If you are just starting to use NICER data, the NICER team highly recommends that you begin your work with the newest NICER software and calibration. This release creates much more streamlined workflows for both spectral and light curve extraction, so it is probably not worth investing time learning older techniques.

Starting a New Project. If you are starting a new project, the NICER team recommends that you update your software and calibration. Although the new energy scale released with this calibration update will provide marginal improvements in analysis, the new automatic screening tools may be beneficial to users.

Working with High Optical Light Loading. NICER experienced an optical light leak on May 22, 2023, which significantly increased the chances of high optical light loading during orbit day, depending on sun angle. The potential for optical light loading has always existed since NICER launched, but the likelihood to extreme conditions has increased significantly after the light leak developed. This release has several capabilities that aid in the process of screening, interpreting and analyzing data taken those circumstances. It is highly recommended to reprocess data observed after May 22 or under high optical loading conditions.

A separate, but related, question is, am I required to reprocess my data to take advantage of these new features? The short answer is yes, you are required to reprocess your data with nicerl2 to use the new screening capabilities This is mostly because certain new filter file columns are required for the new features to work properly.

Potential Incompatibility Gotchas

Generally the NICER team strives to maintain forward and backward compatibility. This improves the user experience with our software and also instills confidence in the results from our observatory. However, in this release, there are the potential for some minor incompatibilities.

Filter File Changes. The filter file (MKF file) columns have changed since the previous release. Some columns have been added or changed (for more information, please see the Filter Files thread). Although it is advised to reprocess all of your data when starting a new project, there may be some situations where you have to merge filter files from across versions. Use the task nimkfmerge or niobsmerge for this situation. For details on how to use these tasks, please see the Filter Files or Combining Observations threads, respectively, for more information.

Two columns in the filter file, MPU_LOWMEM_FIFO_DELTA and MPU_LOWMEM_SCI_DATA, have been upgraded to be 32-bit numbers instead of 8-bit numbers. Under high optical loading conditions, an 8-bit number is not sufficient to store these counts.

For observation data taken under high optical light loading conditions, especially after May 22, 2023, it is highly recommended to reprocess your entire observation with the new release to gain an upgraded filter file.

EVENT_FLAGS changes. The NICER team has discovered a new phenomenon which we call "noise ringers." These are events that occur within a short period of time after an undershoot event (within 110 usec). Under high optical loading conditions, events within that time are almost always noise. NICER now uses an EVENT_FLAGS bit (bit 0x40) to indicate that an event is within that time window. Screening of these events is done automatically in this release. However, users who manually screen data with an explicit expression involving EVENT_FLAGS should be aware that an extra bit is used.

In the rare case that you are using a manually-specified event filtering expression which includes EVENT_FLAGS, you should change to use a full 8-bit bit-matching expression, for example "EVENT_FLAGS == bxxxxxx00". If you don't use a manual EVENT_FLAGS expression, the full eight bit transition is handled automatically for you. Users should also investigate using the keep_* parameters of nicerl2/nicermergeclean/nicerclean instead of using an EVENT_FLAGS expression.

New screening by default. Users should be aware that several screening options are enabled by default.

  • max_lowmem=250 - screens out data larger than max_lowmem lost events per second in the entire array
  • thresh_range=-3-3 - screens data to accept data with a low energy threshold in the nominal night time range (set to 32-38 for light leak orbit day; -3-38 for all night and day thresholds)
  • mingti=5.0 - minimum GTI size of 5 seconds
  • erodedilate=5.0 - remove shredded GTI pieces smaller than 5 seconds
  • keep_noisering=NO - remove noise ringer events when undershoots > noisering_under ct/s/FPM
  • niautoscreen - now has sensible absolute limits for all "bad" data indicators

Running NICER spectra and light curve tools with gtifile option does not work with 3C50 background model. The nicerl3-spect and nicerl3-lc tools can accept a parameter called gtifile, which allows the user to do detailed time filtering on on an existing screened event list. However, this parameter is not compatible with all background models, such as 3C50 (which does not understand the additional time filtering option). When using the 3C50 model you will need to re-run nicerl2 with the gtifiles parameter instead.

Details: Dealing with High Optical Loading

The silicon detectors that NICER uses, which are sensitive to X-rays, are also sensitive to optical light. Thus, the effects of optical light can affect NICER analysis, sometimes strongly so.

The effects of optical light loading include:

  • shift of energy scale (modeled in gain calibration)
  • broadening of resolution (modeled in response calibration)
  • increase of noise peak (modeled in SCORPEON model)
  • (NEW) "noise ringer" events are enhanced

In addition, NICER experienced an optical light leak, which developed on May 22, 2023, and continues to affect operations and scientific analysis. The primary effects of this optical light leak are to increase the effects of optical loading on the detectors. The NICER team is providing new tools and screening in this release to deal with these effects.

The May 22 optical light leak has dramatically increased the amount of optical light that can reach NICER detectors. The NICER team has developed several web pages that discuss what has occured, the operational impacts, and how scientists can deal with the new conditions. We refer scientists to these pages for more information.

Separately, the NICER team has become aware of a phenomenon noted above, which we are calling "noise ringers," that affects the 0.3-0.6 keV energy range of spectra during high optical loading conditions. Essentially, this is a noise peak "tail" that shows up in these conditions which may prevent analysis of soft spectra, or be confused for an X-ray feature. Our page on Noise Ringers has more detail on this phenonemon and how to deal with it from an analysis point of view.

Details: Preventing "Shredded" GTIs

The phenomenon of "shredded GTIs" refers to the possibility of NICER event data to have many small GTIs. The GTIs are typically a fraction of a second long, so if one looks at an exposure versus time plot, it appears that the exposure is divided into thousands of small pieces, or "shredded."

Shredding can be caused by two major factors.

  • On-board, the MPU is unable to transmit all packaged counts, typically due to extreme count rates
  • On the ground, screening on variables that are noisy

On-board, the Measurement Processing Unit (MPU) is responsible for digitizing counts, packaging them, and transmitting them to the ground. Under extreme count rate conditions, it is possible for more counts to be generated by the MPU than are possible to transmit (i.e. the data link rate is smaller than the data production rate). In that case, the MPU will discard packaged events, which leads to an exposure gap. Events are packaged into 100-event groups, and the maximum transmission rate is about 55 packets per second, so shredding due to this cause will have tens of gaps per second.

On the ground, screening can produce shredding effects as well. Some of the screening variables, such as FPM_OVERONLY_COUNT (overshoots) have statistical noise. If distribution of the variable is broader than the range specified for screening, then the result can be shredded GTIs. For example, if overshoots range between 1 and 20 ct/s, and the overonly_range is set to "0-10" then there will be many small GTIs as the overshoots vary back and forth over the "10" threshold.

Shredded GTIs are not beneficial to scientific analysis. They make exposure calculation difficult and inaccurate In addition, the underlying causes (i.e. extreme count rates) would imply that instrument deadtimes are also extreme, again making exposure calculations potentially inaccurate.

This release employs several new techniques to reduce the problems of Shredded GTIs. For noisy screening, now will filter out small GTIs due to noisy variables (overshoots and undershoots). For MPU shredding due to extreme count rates, niautoscreen is now more robust about letting some (or all) MPUs from skewing the allowed range.

For the noisy screening portion of the issue, nicerl2/nimaketime have two new parameters called mingti and erodedilate which will remove shredded GTIs with "shred" size ~5 seconds or smaller. This is enabled by default.

In addition, the autoscreening tool, niautoscreen, enforces some sanity checks, which prevent many MPUs with shredding from skewing the screening. Previously, users sometimes reported that after autoscreening, only about 8 detectors (one MPU's worth) were left selected, and this is due to that skewing. The new algorithm enforces stronger sanity constraints. However, a caveat to this enhanced treatment is that users may find no good time after running niautoscreen rather than ~8 detectors. This is the desired behavior to prevent heavily "shredded" MPUs from biasing your spectral analysis.

Details: No Calibration Update

HEASoft 6.32 / NICERDAS 11 does not involve published calibration changes. The existing published NICER calibration remains in effect.

NICER analysis should continue to use NICER CALDB release xti20221001, which was released in November, 2022. Please see the NICER's Calibration Recommendations for current best practice, and NICER Calibration Documents for calibration release notes.

For example, the required auxiliary files for background generation, and bad times for automatic screening, are included in CALDB.

Also, this release has a new energy scale for both the slow (PI) and fast (PI_FAST) channels. The updates improve the cross-alignment of the energy scale between detectors, especially at high energies above 8 keV. This change has negligible effect below 8 keV (< 5 eV on average). Above 8 keV there are large effects, which increase to a maximum of ~300 eV for some detectors at 13 keV. In practice, few scientists will notice a change, since most NICER science counts are in the 0.3 - 6 keV range.

Complete Change Log

  • niobsermerge (NEW) to combine observation data sets
    • nicerl3-spect and nicerl3-lc updated to work with merged data sets
    • nimkfmerge now properly sets TSTART/TSTOP/DATE-OBS/DATE-END in merged outputs
  • nicerl2
    • update to filtcols=NICERV5 (see niprefilter below for new columns)
    • new parameters mingti/erodedilate to avoid shredded GTIs
    • new parameters max_lowmem and thresh_range to deal with light leak situation
    • new parameters keep_{forced,overshoots,undershoots}, datamode for event filtering
    • new parameter keep_noisering, noisering_under for filtering noise ringer events
    • improved default verbosity with more user-helpful output
    • bug fix to properly pass incremental parameter to prep_mkfiles
    • add geomag_tcheck parameter, allow dealing with stale geomag data
  • nicerl3-spect / nicerl3-lc
    • bug fix to not accept gtifile when 3C50 background model is used
    • ignore ufafile/indir/cldir parameters if not needed
    • handle EXPOSURE=0 situations more gracefully by returning status=218
    • improved default verbosity
    • clean up temporary tco plot files
  • nicerl3-lc
    • allow renormalization of light curve by number of selected detectors
    • new parameters detnormtype/detnormbuff/detnormchg to accomodate normalization
    • default is detnormtype=arr52, which scales light curve to equivalent 52-detector array
  • niprefilter / niprefilter2
    • new NICERV5 filter columns for filter file (below)
    • new columns {FPM,MPU}_NOISERING_COUNTS to diagnose noise ringer events
    • new column SUN_BODY_AZIMUTH to azimuth of sun in NICER body coordinates
    • new columns TOT_LOWMEM_{SCI,FIFO} for total array to diagnose lost events
    • new column DELTA_SLOW_LLD to diagnose array-averaged low energy threshold changes
    • bug fix to handle event files with no rows
  • niautoscreen
    • better graphical display of censored detectors and MPUs with default chatter
    • bug fix for round-robin detection (was more agressive disabling than should be)
    • parameter limits now include min/max as well for overall sanity checking
  • nimpumerge
    • new parameter fpmsel (=YES by default) to properly manage FPM selection data
  • nibackgen3C50: (v1.2) fixed CEATOR keyword in output (should be CREATOR)
  • nicerarf & nicerrmf: handle failures more gently by passing status=218 up to the caller
  • nicerclean
    • added keep_{forced,overshoots,undershoots,noisering} and data mode keywords;
    • dded noisering_under parameter for noise ringer filtering support
  • nimaketime
    • calculates and prints how filtering stages impact available good time
    • add min_lowmem and thresh_range parameters for dealing with light leak situation
    • add mingti and erodedilate parameters for "smoothing" shredded GTIs
  • nicerl2 / nicermergeclean / nicerclean
    • remove excplicit parameters from par file, use DEFAULT instead
    • for example roundrobbinscr
    • new parameters added keep_{undershoots,overshoots,forced,noisering} to match nicerclean
    • new parameters datamode noisering_under parameters to match nicerclean
  • nifpmsel
    • handle case of full MPU disabled better
    • handle tricky case where all FPMs in an MPU are GTI-selected by same GTI
  • nigeodown
    • Removed Perl LWP::Simple requirement (was not used anyway)
    • accomodate relative paths
  • niprescreen (NEW) tool for pre-screening NICER event data
    • not meant to be called by user
    • nimpucal updated to call niprescreen
    • new EVENT_FLAGS bit 0x40 for 'noise ringer' events detected
  • nicer-l1-metadata: fixed excludepat= Perl (>v5.10) handling and set default to excludepat=NONE
  • niextract-events: Write revised DATE-OBS, DATE-END, and (optionally) OBS_ID keywords
  • nimpucal: optimized calculation of PI_RATIO
  • nivigsum: Better handling of off-axis warning when large numbers of detectors are disabled
    • niscorpcalc: Bug fix to properly read SAA/trapped map metadata keywords
    • internally factor code into Perl library
  • Multiple tasks updated to change overall chatter level and logging to provide more useful information to the user and less chaff

Related Topics

For instructions on setting up a NICER analysis environment, please see the Setting Up a NICER Analysis Environment thread.

For more information on recommended settings and calibration recommendations, please see the NICER Calibration Recommendations analysis thread.

For release notes for the previous release (HEASoft 6.31 / NICERDAS 10), please see the Software Release HEASoft 6.31 thread.


  • 2023-07-11 - initial draft
  • 2023-07-21 - new subsection on shredded GTIs; add discussion of new niautoscreen behavior