Analysis Recommendations After the Light Leak

Overview

Starting on May 22, 2023, NICER experienced a light leak that significantly affects operations and scientific analysis. This page discusses recommendations to observers on how to handle the effects of the light leak when analyzing their data.

Read this thread if you want to: Properly analyze NICER data affected by the light leak

Last update: 2024-01-25

Introduction

After May 22, 2023, NICER is affected by an optical light leak. Damage to one or more of NICER's thermal films, which are also optical blocking filters, has allowed a significant amount of additional optical light to enter the NICER detectors.

The impacts of the optical light leak are significant, and have forced significant changes to how NICER operations, and also to the nature of NICER's data and how it should be processed. For an overview of what happened and the impacts, please see the Light Leak Overview page.

As a summary, NICER's performance is unchanged during orbit night, but during orbit day, NICER observes targets at larger angles from the sun and earth limb. NICER data volumes are higher, and the chances of data-rate related artifacts has increased.

This page discusses the approach observers should take when analyzing data affected by the light leak.

Overall Recommendations

Overall, NICER users will benefit by obtaining the most recent NICER software release as of this writing, HEASoft 6.32 (NICERDAS 11), released in July 2023. This release contains several improvements that are designed specifically to improve screening or performance for data affected by the optical light leak.

If you have any data taken after the May 22, 2023 onset of the optical light leak, it is recommended to reprocess all of the NICER data using HEASoft 6.32 or higher. This release contains changes to the filter file that are best realized if all of the data in your observation have the same columns and formats.

Please be aware that the default screening does the following:

  • Keeps the standard undershoot range of 0-500 ct/s/FPM
  • Adds a new screeng criterium which excludes times when lost events exceeds 250 lost events per second
  • Automatically screens out noise ringer events (within 110 usec of an undershoot)
  • Automatically selects only nominal-threshold data
For daytime threshold events, please see the discussion below on commanded threshold changes in order to understand how to process daytime data.

Dealing with High Undershoot Rates

Data taken during orbit day after the light leak have significantly larger numbers of events. Most of these events are undershoots, representing the high optical loading conditions during orbit day. File sizes may have grown by a factor of ten or more, compared to before the light leak.

For the most part, the impact for these high event rates is download and computing time. These are practical issues that will affect all users, but do not necessarily affect the overall quality of the data. Data taken at high undershoot rates can still be useful.

The NICER calibration is valid for undershoot rates in the 0-500 ct/s/FPM range. Response and energy scale calibrations are designed to work throughout that range.

Users should maintain the default undershoot range (underonly_range=0-500) when processing data. Despite temptations, do not try to extend the range beyond 500 ct/s/FPM, even if that results in no "good" data after running nicerl2.

To reiterate, undershoots above 500 ct/s/FPM are an indication of very low quality observation data. Above this limit, the energy scale becomes highly distorted, unmodeled dead-times become extreme, and in fact the count of undershoots themselves become unreliable. NICER cannot vouch for the calibration of NICER data outside of the quoted undershoot range. For all these reasons, the NICER team does not recommend extending the undershoot range higher than 500 ct/s/FPM.

Lost Events

As described on the Light Leak Overview page, lost events occur when an MPU becomes so busy that some (or many) events are discard before they are even packetized.

The impacts of lost events are particularly insidious:

  • Lost X-rays means the total effective area throughput becomes invalid. In other words, the flux scale becomes uncalibrated
  • Lost undershoots means that the undershoot count rate is underestimated, and therefore the energy scale shifts are not correct.
Thus, lost events create serious calibration problems and should be avoided.

As of HEASoft 6.32 (NICERDAS 11), released in July 2023, there are new diagnostics to avoid periods of time where lost events occur.

First, there is a new filter file column called TOT_LOWMEM_FIFO which captures the rate of lost events, per second, for the entire NICER array (all seven MPUs). This column is a sum of the pre-existing MPU_LOWMEM_FIFO_DELTA column.

However, in versions of NICER software before HEASoft 6.32, these columns were not totaled, and do not have enough bits to store the true magnitude of lost events during orbit day. Therefore, it is recommended to reprocess your data, in entirety, if they contain data observed during the light leak.

HEASoft 6.32 also has a new default screening criterium to avoid lost events: lost events must be less than 250 events/second to be considered good. This magnitude is somewhat arbitrary and chosen to accomodate the possibility of this column being stored as an 8-bit number. However, any number of lost events is a bad situation, and should be avoided.

The default HEASoft 6.32 screening of "max_lowmem=250" should be used for NICER observations that overlap with the optical light leak.

Lost Packets

NICER may also experience lost packets under high event rate situations. Unlike lost events, however, lost packets are properly accounted for by good time intervals (GTIs) that are derived from the packet data itself. Since packets contain enough information to determine when some packets are missing, the NICER software uses this to assign correct exposure time.

However, if many packets are lost, this can lead to a "GTI shredding" appearance to your data. Here, "shredding" means that there are many small GTIs corresponding to many individual lost packets. In principle, however, data should still be good in this situation; it just has less than 100% exposure efficiency.

No special action is required to deal with lost packets.

Noise Ringer Events

Starting in early 2023, the NICER team became aware of a phenomenon that we are calling "noise ringers."

Noise ringers are a particular detector phenomenon that manifest preferentially during times of high undershoot levels. The greatest symptom of noise ringers from a science analysis point of view is to create additional noise in the 0.2-0.6 keV pulse height range. This noise is outside of the normal range of the noise peak, but interactions between undershoot resets and the noise peak tends to produce a visible tail that extends to higher energies.

NICER spectra can experience noise ringers at any time there are high optical loading conditions, typically above about 80 ct/s/FPM. This is true both after and before the optical light leak of May 22, 2023. However, since optical loading has become such a greater effect after May 22, the chances of experiencing noise ringers is higher.

For more information on noise ringers, please see the Noise Ringers documentation page.

Starting in HEASoft 6.32, the standard NICER tools can recognize and screen out noise ringer events. This screening is automatic when the keep_noisering=YES parameter is set when calling nicerl2, which is recommended. This operation removes all events in an FPM that occur within 110 usec of an undershoot in the same FPM, but only if the undershoot rate is greater than 80 ct/s/FPM (this value is controled by the noisering_under parameter).

The standard filtering will have no effect when undershoots are low (below 80 ct/s/FPM), and only when undershoots exceed that threshold does the removal begin.

The impact of using this screening is to create a small amount of additional deadtime around undershoot events. Typically the deadtime is very small (~few percent), and at the maximum calibrated undershoot rate of 500 ct/s/FPM, the maximum dead time is ~5%.

The NICER team recommends to use the default filtering of HEASoft 6.32 when processing data that overlaps the optical light leak. This will remove noise ringer events automatically, when undershoot rates are high.

Commanded Threshold Changes

As of June 2, 2023, the NICER team routinely adjusts the low energy threshold during orbit day observations.

The impact of this operational change is to change the effective energy range over which NICER observes X-ray targets. Before the light leak, and during all orbit night observations, the X-ray threshold is set to be effectively ~250 eV, although the shape of the trigger efficiency curve is more complicated than a simple sharp cutoff at low energies (see FPM Detector and XRC parameters for more details).

During orbit day after the light leak, the low energy threshold has been increased by approximately 150 eV to a new centroid of about 400 eV.

The figure below shows threshold variations as a function of time.

Figure. Plot of NICER array-averaged threshold setting (filter file column DELTA_SLOW_LLD) for 20000 seconds at the beginning of UTC day 2023-07-09.

As the plot shows, the threshold steps up and down, corresponding to transitions between orbit day and night, respectively. HEASoft 6.32 adds a new column to the filter file called DELTA_SLOW_LLD which aids in tracking array-averaged threshold adjustments such as this.

As of HEASoft 6.32, nicerl2 now has a new parameter, thresh_range which governs which thresholds are considered good when screening data by time.

It is the NICER team recommendation to separate data taken with different thresholds. The thresh_range parameter can help do this.

Screening During Orbit Night

The default thresh_range is -3.0 to 3.0, which basically corresponds to a range that includes zero. Thus the default value of thresh_range can be used to process data taken during orbit night.

The default settings of nicerl2 will likely exclude data taken during orbit day. You will have to make a conscious and explicit change when running nicerl2 to include data taken with orbit day thresholds.

After running nicerl2 with its default settings, you can use all standard processing to create spectral and light curve products, as described on our Analysis Threads pages.

Screening During Orbit Day

Data taken during orbit day may present some additional challenges, and the threshold setting is one of those challenges.

NICER software was designed to work with data taken with nominal thresholds. Trying to mix and match data taken with orbit night and day thresholds may lead to significant calibration problems that the NICER team will be unable to support at this time. We are working on solutions to combine orbit night and day data, but it may take some time to gain this capability.

However, it is still possible to use NICER data taken with orbit day thresholds. There may be significant excellent and usable data, so the NICER team makes available the possibility to choose a different threshold range.

The default threshold setting for orbit day is currently "+35". The units of this value are arbitrary, as it represents the setting that the operations team uses to adjust thresholds. The actual value in eV is more difficult to estimate and is a topic being investigated.

To process data taken with orbit day thresholds, you can use the following settings to nicerl2.

  • orbit day only - thresh_range=32-38
  • orbit day and night combined - thresh_range=-3-38 (not recommended)

As noted above, the NICER team does not recommend to mix data taken with orbit day and night thresholds. Instead we recommend extracting a "night" spectrum, followed by a "day" spectrum, here referring to the night or day threshold setting.

This can be accomplished by running nicerl2 first with the default threshold range, which corresponds to the night threshold setting: nicerl2 NNNNNNNNNN ... clobber=YES and extracting any desired products and/or light curves. Then, re-run nicerl2 with the day threshold range: nicerl2 NNNNNNNNNN ... thresh_range=32-38 clobber=YES and again extract any desired products.

As a side note, if you are experimenting with various screening parameters and re-running nicerl2 multiple times, you do not have to wait for nicerl2 to completely re-process everything from the raw data. You can use the "tasks" and "incremental" parameters of nicerl2 to avoid lengthy reprocessing. Please see the Making nicerl2 Run Faster section of the nicerl2 page.

For example after you have run nicerl2 once completely for the orbit night threshold settings, you do not need to run the calibration and/or filter file steps again. When running nicerl2 again with the orbit day threshold setting, you can use the tasks parameter to limit processing to just the screening portion of nicerl2. Example: nicerl2 NNNNNNNNNN ... thresh_range=32-38 tasks=SCREEN clobber=YES

One thing to realize is that, in addition to the thresh_range parameter, there are other settings to nicerl2 which affect how data are screened. These include the underonly_range parameter, which governs the allowed range of undershoots, and max_lowmem, which governs how many lost events are allowed. Since data taken after the light leak during orbit day can have very high undershoot rates (>500 ct/s/FPM) and very high lost event rates (>250 lost events/s), these other screening criteria may remove all or most of the data regardless of the thresh_range setting. Therefore, users should be prepared for "no data" or "zero exposure" being an expected outcome for data taken after the light leak during orbit day.

When using observation data taken with orbit day thresholds, please be aware of the following caveats:

  • No data may survive the underonly_range, max_lowmem and thresh_range settings.
  • The response matrix is not calibrated at low energies; the response for photon energies from approximately 250-400 eV will be incorrect. The exact response changes are still under analysis.
  • None of the background models provides usable estimates at low energies. All will still provide a background assuming the threshold setting corresponds to orbit night.

Given these caveats, it makes sense to treat the lowest energies (around 400 eV and below) with special care. It may be wise to ignore all energies below 500 eV.

Modifications

  • 2023-07-14 - initial draft
  • 2024-01-25 - note that "no data" is a possibility after the standard screening; not recommended to extend undershoot range above 500