Analysis Recommendations After the Light LeakReturn to: Analysis Threads | Analysis Main Page
OverviewStarting on May 22, 2023, NICER experienced a light leak that significantly affects operations and scientific analysis. This page discusses recommendations to observers on how to handle the effects of the light leak when analyzing their data. Read this thread if you want to: Properly analyze NICER data affected by the light leak Last update: 2024-02-16 IntroductionAfter May 22, 2023, NICER is affected by an optical light leak. Damage to one or more of NICER's thermal films, which are also optical blocking filters, has allowed a significant amount of additional optical light to enter the NICER detectors. The impacts of the optical light leak are significant, and have forced significant changes to how NICER operations, and also to the nature of NICER's data and how it should be processed. For an overview of what happened and the impacts, please see the Light Leak Overview page. As a summary, NICER's performance is unchanged during orbit night, but during orbit day, NICER observes targets at larger angles from the sun and earth limb. NICER data volumes are higher, and the chances of data-rate related artifacts has increased. This page discusses the approach observers should take when analyzing data affected by the light leak. Overall RecommendationsOverall, NICER users will benefit by obtaining the most recent NICER software release as of this writing, HEASoft 6.32 (NICERDAS 11), released in July 2023. This release contains several improvements that are designed specifically to improve screening or performance for data affected by the optical light leak. If you have any data taken after the May 22, 2023 onset of the optical light leak, it is recommended to reprocess all of the NICER data using HEASoft 6.32 or higher. This release contains changes to the filter file that are best realized if all of the data in your observation have the same columns and formats. Please be aware that the default screening does the following:
Dealing with High Undershoot RatesData taken during orbit day after the light leak have significantly larger numbers of events. Most of these events are undershoots, representing the high optical loading conditions during orbit day. File sizes may have grown by a factor of ten or more, compared to before the light leak. For the most part, the impact for these high event rates is download and computing time. These are practical issues that will affect all users, but do not necessarily affect the overall quality of the data. Data taken at high undershoot rates can still be useful. The NICER calibration is valid for undershoot rates in the 0-500 ct/s/FPM range. Response and energy scale calibrations are designed to work throughout that range. Users should maintain the default undershoot range (underonly_range=0-500) when processing data. Despite temptations, do not try to extend the range beyond 500 ct/s/FPM, even if that results in no "good" data after running nicerl2. To reiterate, undershoots above 500 ct/s/FPM are an indication of very low quality observation data. Above this limit, the energy scale becomes highly distorted, unmodeled dead-times become extreme, and in fact the count of undershoots themselves become unreliable. NICER cannot vouch for the calibration of NICER data outside of the quoted undershoot range. For all these reasons, the NICER team does not recommend extending the undershoot range higher than 500 ct/s/FPM. Lost EventsAs described on the Light Leak Overview page, lost events occur when an MPU becomes so busy that some (or many) events are discard before they are even packetized. The impacts of lost events are particularly insidious:
As of HEASoft 6.32 (NICERDAS 11), released in July 2023, there are new diagnostics to avoid periods of time where lost events occur. First, there is a new filter file column called TOT_LOWMEM_FIFO which captures the rate of lost events, per second, for the entire NICER array (all seven MPUs). This column is a sum of the pre-existing MPU_LOWMEM_FIFO_DELTA column. However, in versions of NICER software before HEASoft 6.32, these columns were not totaled, and do not have enough bits to store the true magnitude of lost events during orbit day. Therefore, it is recommended to reprocess your data, in entirety, if they contain data observed during the light leak. HEASoft 6.32 also has a new default screening criterium to avoid lost events: lost events must be less than 250 events/second to be considered good. This magnitude is somewhat arbitrary and chosen to accomodate the possibility of this column being stored as an 8-bit number. However, any number of lost events is a bad situation, and should be avoided. The default HEASoft 6.32 screening of "max_lowmem=250" should be used for NICER observations that overlap with the optical light leak. Lost PacketsNICER may also experience lost packets under high event rate situations. Unlike lost events, however, lost packets are properly accounted for by good time intervals (GTIs) that are derived from the packet data itself. Since packets contain enough information to determine when some packets are missing, the NICER software uses this to assign correct exposure time. However, if many packets are lost, this can lead to a "GTI shredding" appearance to your data. Here, "shredding" means that there are many small GTIs corresponding to many individual lost packets. In principle, however, data should still be good in this situation; it just has less than 100% exposure efficiency. No special action is required to deal with lost packets. Noise Ringer EventsStarting in early 2023, the NICER team became aware of a phenomenon that we are calling "noise ringers." Noise ringers are a particular detector phenomenon that manifest preferentially during times of high undershoot levels. The greatest symptom of noise ringers from a science analysis point of view is to create additional noise in the 0.2-0.6 keV pulse height range. This noise is outside of the normal range of the noise peak, but interactions between undershoot resets and the noise peak tends to produce a visible tail that extends to higher energies. NICER spectra can experience noise ringers at any time there are high optical loading conditions, typically above about 80 ct/s/FPM. This is true both after and before the optical light leak of May 22, 2023. However, since optical loading has become such a greater effect after May 22, the chances of experiencing noise ringers is higher. For more information on noise ringers, please see the Noise Ringers documentation page. Starting in HEASoft 6.32, the standard NICER tools can recognize and screen out noise ringer events. This screening is automatic when the keep_noisering=YES parameter is set when calling nicerl2, which is recommended. This operation removes all events in an FPM that occur within 110 usec of an undershoot in the same FPM, but only if the undershoot rate is greater than 80 ct/s/FPM (this value is controled by the noisering_under parameter). The standard filtering will have no effect when undershoots are low (below 80 ct/s/FPM), and only when undershoots exceed that threshold does the removal begin. The impact of using this screening is to create a small amount of additional deadtime around undershoot events. Typically the deadtime is very small (~few percent), and at the maximum calibrated undershoot rate of 500 ct/s/FPM, the maximum dead time is ~5%. The NICER team recommends to use the default filtering of HEASoft 6.32 when processing data that overlaps the optical light leak. This will remove noise ringer events automatically, when undershoot rates are high. Commanded Threshold ChangesAs of June 2, 2023, the NICER team routinely adjusts the low energy threshold during orbit day observations. The impact of this operational change is to change the effective energy range over which NICER observes X-ray targets. Before the light leak, and during all orbit night observations, the X-ray threshold is set to be effectively ~250 eV, although the shape of the trigger efficiency curve is more complicated than a simple sharp cutoff at low energies (see FPM Detector and XRC parameters for more details). During orbit day after the light leak, the low energy threshold has been increased by approximately 150 eV to a new centroid of about 400 eV. The figure below shows threshold variations as a function of
time.
As the plot shows, the threshold steps up and down, corresponding to transitions between orbit day and night, respectively. HEASoft 6.32 adds a new column to the filter file called DELTA_SLOW_LLD which aids in tracking array-averaged threshold adjustments such as this. As of HEASoft 6.32, nicerl2 now has a new parameter, thresh_range which governs which thresholds are considered good when screening data by time. In HEASoft 6.33 and higher, the threshfilter paramter is preferred for controlling threshold-related selections, although thresh_range continues to work. It is the NICER team recommendation to separate data taken with different thresholds. The threshfilter parameter can help do this. Screening During Orbit NightThe default threshfilter is "NIGHT", which corresponds to the "orbit night" threshold range. This is equivalent to thresh_range of -3.0 to 3.0, which basically corresponds to a range that includes zero. Thus the default values for the threshold setting can be used to process data taken during orbit night, or for any data before May 2023, which uses the "night" threshold setting. The default settings of nicerl2 will likely exclude data taken during orbit day. You will have to make a conscious and explicit change when running nicerl2 to include data taken with orbit day thresholds. After running nicerl2 with its default settings, you can use all standard processing to create spectral and light curve products, as described on our Analysis Threads pages. Screening During Orbit DayData taken during orbit day may present some additional challenges, and the threshold setting is one of those challenges. NICER software was designed to work with data taken with nominal thresholds. Trying to mix and match data taken with orbit night and day thresholds may lead to significant calibration problems that the NICER team will be unable to support at this time. We are working on solutions to combine orbit night and day data, but it may take some time to gain this capability. However, it is still possible to use NICER data taken with orbit day thresholds. There may be significant excellent and usable data, so the NICER team makes available the possibility to choose a different threshold range. The default threshold setting for orbit day is currently "+35", in analog to digital units (ADUs). The units of this value are arbitrary, as it represents the setting that the operations team uses to adjust thresholds. This shift corresponds to changing the default threshold of about 0.25 keV to about 0.38 keV. In reality the low energy trigger efficiency function is a more complicated curve than just a sharp cutoff, but the centroid of this curve shifts by the indicated amount. To process data taken with orbit day thresholds, you can use the following settings to nicerl2.
As noted above, the NICER team does not recommend to mix data taken with orbit day and night thresholds. Instead we recommend extracting a "night" spectrum, followed by a "day" spectrum, here referring to the night or day threshold setting. As detailed below, the software released with HEASoft 6.33 (NICERDAS 12) has support for analyzing orbit night and orbit day products separately.
This can be accomplished by running nicerl2 first with the default
threshold range, which corresponds to the night threshold setting:
As a side note, if you are experimenting with various screening parameters and re-running nicerl2 multiple times, you do not have to wait for nicerl2 to completely re-process everything from the raw data. You can use the "tasks" and "incremental" parameters of nicerl2 to avoid lengthy reprocessing. Please see the Making nicerl2 Run Faster section of the nicerl2 page.
For example after you have run nicerl2 once completely for the orbit night
threshold settings, you do not need to run the calibration and/or filter file
steps again. When running nicerl2 again with the orbit day threshold setting,
you can use the tasks parameter to limit processing to just the
screening portion of nicerl2. Example:
One thing to realize is that, in addition to the threshfilter parameter, there are other settings to nicerl2 which affect how data are screened. These include the underonly_range parameter, which governs the allowed range of undershoots, and max_lowmem, which governs how many lost events are allowed. Since data taken after the light leak during orbit day can have very high undershoot rates (>500 ct/s/FPM) and very high lost event rates (>250 lost events/s), these other screening criteria may remove all or most of the data regardless of the thresh_range setting. Therefore, users should be prepared for "no data" or "zero exposure" being an expected outcome for data taken after the light leak during orbit day. When using observation data taken with orbit day thresholds, please be aware of the following caveats:
Given these caveats, it makes sense to treat the lowest energies (around 400 eV and below) with special care. NICER Software Support for Data Taken With Light LeakAs of HEASoft 6.33 / NICERDAS 12, the standard NICER tools do have support for dealing with most aspects of data taken after May 2023. We strongly recommend updating your software to the most recent version to take advantage of these changes.Here we discuss NICER's support for data taken under optical light leak conditions. The following table shows the support for various analysis activities based on the observation conditions.
As can be seen from this table, NICER supports analysis of essentially all combinations of observing conditions. The place where support and analysis becomes challenging is for users who require information below 0.5 keV with data taken during orbit day conditions. The primary challenge here is that the low energy threshold setting during orbit day is changed to reduce noise levels. The response modeling tools support analysis of data under all conditions, beginning with HEASoft 6.33. The SCORPEON background model supports analysis of data under all conditions, beginning with HEASoft 6.33. However, it should be made clear that SCORPEON can only predict a background for one threshold setting, either "night" or "day," but not a mix of settings. Thus users will need to extract products separately for orbit day and orbit night. The nicerl2 page has more useful information on how to do this. The 3C50 and Space Weather background models do not support analysis during orbit day after May 2023. The primary reason is that these background models are "library" models and the libraries were accumulated during pre-2023 (or orbit night) conditions. Changing to a different threshold setting is non-trivial. Therefore, the NICER team currently does not recommend analyzing data below 0.5 keV for data taken after May 2023 during orbit day conditions. In those situations, users of the 3C50 or Space Weather models should extract a night-only spectrum or ignore data below 0.5 keV. For data taken after the optical light leak, users should keep data observed during orbit night separate from data observed during orbit day. The nicerl2 page discusses how to do this in more detail using HEASoft 6.33. The SCORPEON model allows background modeling of orbit day data below 0.5 keV; the 3C50 and Space Weather models do not. Modifications
|
- HEASARC Director: Dr. Andy Ptak
- NASA Official: Dr. Andy Ptak
- Web Curator: J.D. Myers
- Last Modified: 16-Feb-2024