Software Release HEASoft 6.32 / NICERDAS 11Return to: Analysis Threads | Analysis Main Page
OverviewThis page discusses new changes available in NICERDAS version 11a, distributed in HEASoft 6.32.1. This release is an intermediate NICER software and calibration release, with signifigant improvements. Users will find many of the changes beneficial, especially for data taken after May 2023. Please note that HEASoft 6.32 / NICERDAS contains a bug that results in incorrect normalization and screening of data. Please upgrade all installations to HEASoft 6.32.1, which was released on August 23, 2023. Read this thread if you want to: Understand NICERDAS 11 software Last update: 2023-08-23 IntroductionObtaining useful scientific results from NICER observations is a combination of NICER observational data with software and calibration products. Over time, NICER is committed to improving software and calibration so that scientists can get the most out of their data. This page discusses changes available in NICERDAS 11a, released with HEASoft 6.32.1 in August 23, 2023. A previous release of July 2023, HEASoft 6.32 / NICERDAS 11, contains a bug.
HEASoft 6.32.1 is an intermediate NICER software feature release.
While no major calibration or workflow changes are provided, there are
significant accomodations for data taken after May 2023, when NICER
developed an optical light leak. There are other beneficial
improvements that are worth checking out, as detailed in this article.
As always, we recommend users to document which version of software they are using, especially for publication. This will help future analysts repeat your results. For more information, see the Reporting NICER software and calibration versions thread. HEASoft 6.32.1 Patch Release Fixes Incorrect NormalizationThe original HEASoft 6.32 release contains a bug which results in incorrect normalization in many cases. The bug occurs in the task 'nifpmsel', and impacts data where per-MPU screening is requested by the user or by automatic screening. The bug affects only HEASoft 6.32, and not earlier or later released versions. More information about the bug can be found on the MPU Autoscreening Bug page. The only remedy is to upgrade to HEASoft 6.32.1, which contains NICERDAS 11a. NICERDAS 11a / HEASoft 6.32.1 is available from the HEASoft download page. If you need help installing your NICER analysis environment, please see the NICER Setup page. Overview of NICER ImprovementsHEASoft 6.32 / 6.32.1 (NICERDAS 11 / 11a) is an intermediate NICER software release. This section provides an overview of the changes. More detail is provided in subsequent sections. Standard "Level 3" light curve tools now correct for number of selected / enabled detectors. The task nicerl3-lc has been improved to renormalize a light curve based on the number of selected detectors. This handles the case when users are examining a data set where the number of detectors may change from time to time. This allows light curves to more accurately reflect the flux variations of the X-ray source, rather than variations due to detector turn on-offs. See the nicerl3-lc page for more details. New screening for high optical loading (undershoot) conditions. The nicerl2 task and its subordinate tasks have been improved to deal with conditions that may be seen under very high optical lighting conditions. The potential exists for such situations to occur at any time during NICER's history, but optical light loading is much more prevalent after May 2023, when an optical light leak developed. The filter file has been augmented with new columns (NICERV5) in order to aid this screening. More details can be found below in the "Dealing with High Optical Loading" subsection. New way to combine observation data with the niobsmerge task. Merging observation data has been a commonly-requested functionality which is included in this release. The task niobsmerge combines filter files, event files (both UFA and CL files), as well as the orbit file if available. This allows user to more flexibly work with data that spans many observation segments (many days). More detail on this new feature can be found on our Combining Multiple Observations page. Avoiding "shredded" GTIs. NICER event data can ocassionally have many small GTIs, known as "GTI shredding." These are symptoms of either extreme count rates, or screening on a noisy parameter. This release employs several new techniques to alleviate the problems of shredded GTIs. For more information on this, please see the Preventing Shredded GTIs subsection below. Calibration Updates?This release does not have a companion calibration release. The NICER team continues to work on calibration issues, especially related to the optical light leak that developed in May 2023, but they are not mature enough to distribute at this time. Should I Reprocess My Data?Users will want to know, should they reprocess my data with this new release? The answer depends on the circumstances. Users new to NICER or starting a new project will likely benefit from reprocessing their data with the newest release. In addition, users working with data taken after the May 2023 optical light leak, or working with data with very high undershoot levels, will obtain significant benefits from reprocessing with this release. Users New to NICER. If you are just starting to use NICER data, the NICER team highly recommends that you begin your work with the newest NICER software and calibration. This release creates much more streamlined workflows for both spectral and light curve extraction, so it is probably not worth investing time learning older techniques. Starting a New Project. If you are starting a new project, the NICER team recommends that you update your software and calibration. Although the new energy scale released with this calibration update will provide marginal improvements in analysis, the new automatic screening tools may be beneficial to users. Working with High Optical Light Loading. NICER experienced an optical light leak on May 22, 2023, which significantly increased the chances of high optical light loading during orbit day, depending on sun angle. The potential for optical light loading has always existed since NICER launched, but the likelihood to extreme conditions has increased significantly after the light leak developed. This release has several capabilities that aid in the process of screening, interpreting and analyzing data taken those circumstances. It is highly recommended to reprocess data observed after May 22 or under high optical loading conditions. A separate, but related, question is, am I required to reprocess my data to take advantage of these new features? The short answer is yes, you are required to reprocess your data with nicerl2 to use the new screening capabilities This is mostly because certain new filter file columns are required for the new features to work properly. Potential Incompatibility GotchasGenerally the NICER team strives to maintain forward and backward compatibility. This improves the user experience with our software and also instills confidence in the results from our observatory. However, in this release, there are the potential for some minor incompatibilities. Filter File Changes. The filter file (MKF file) columns have changed since the previous release. Some columns have been added or changed (for more information, please see the Filter Files thread). Although it is advised to reprocess all of your data when starting a new project, there may be some situations where you have to merge filter files from across versions. Use the task nimkfmerge or niobsmerge for this situation. For details on how to use these tasks, please see the Filter Files or Combining Observations threads, respectively, for more information. Two columns in the filter file, MPU_LOWMEM_FIFO_DELTA and MPU_LOWMEM_SCI_DATA, have been upgraded to be 32-bit numbers instead of 8-bit numbers. Under high optical loading conditions, an 8-bit number is not sufficient to store these counts. For observation data taken under high optical light loading conditions, especially after May 22, 2023, it is highly recommended to reprocess your entire observation with the new release to gain an upgraded filter file. EVENT_FLAGS changes. The NICER team has discovered a new phenomenon which we call "noise ringers." These are events that occur within a short period of time after an undershoot event (within 110 usec). Under high optical loading conditions, events within that time are almost always noise. NICER now uses an EVENT_FLAGS bit (bit 0x40) to indicate that an event is within that time window. Screening of these events is done automatically in this release. However, users who manually screen data with an explicit expression involving EVENT_FLAGS should be aware that an extra bit is used. In the rare case that you are using a manually-specified event filtering expression which includes EVENT_FLAGS, you should change to use a full 8-bit bit-matching expression, for example "EVENT_FLAGS == bxxxxxx00". If you don't use a manual EVENT_FLAGS expression, the full eight bit transition is handled automatically for you. Users should also investigate using the keep_* parameters of nicerl2/nicermergeclean/nicerclean instead of using an EVENT_FLAGS expression. New screening by default. Users should be aware that several screening options are enabled by default.
Running NICER spectra and light curve tools with gtifile option does not work with 3C50 background model. The nicerl3-spect and nicerl3-lc tools can accept a parameter called gtifile, which allows the user to do detailed time filtering on on an existing screened event list. However, this parameter is not compatible with all background models, such as 3C50 (which does not understand the additional time filtering option). When using the 3C50 model you will need to re-run nicerl2 with the gtifiles parameter instead. Details: Dealing with High Optical LoadingThe silicon detectors that NICER uses, which are sensitive to X-rays, are also sensitive to optical light. Thus, the effects of optical light can affect NICER analysis, sometimes strongly so. The effects of optical light loading include:
In addition, NICER experienced an optical light leak, which developed on May 22, 2023, and continues to affect operations and scientific analysis. The primary effects of this optical light leak are to increase the effects of optical loading on the detectors. The NICER team is providing new tools and screening in this release to deal with these effects. The May 22 optical light leak has dramatically increased the amount of optical light that can reach NICER detectors. The NICER team has developed several web pages that discuss what has occured, the operational impacts, and how scientists can deal with the new conditions. We refer scientists to these pages for more information.
Separately, the NICER team has become aware of a phenomenon noted above, which we are calling "noise ringers," that affects the 0.3-0.6 keV energy range of spectra during high optical loading conditions. Essentially, this is a noise peak "tail" that shows up in these conditions which may prevent analysis of soft spectra, or be confused for an X-ray feature. Our page on Noise Ringers has more detail on this phenonemon and how to deal with it from an analysis point of view. Details: Preventing "Shredded" GTIsThe phenomenon of "shredded GTIs" refers to the possibility of NICER event data to have many small GTIs. The GTIs are typically a fraction of a second long, so if one looks at an exposure versus time plot, it appears that the exposure is divided into thousands of small pieces, or "shredded." Shredding can be caused by two major factors.
On-board, the Measurement Processing Unit (MPU) is responsible for digitizing counts, packaging them, and transmitting them to the ground. Under extreme count rate conditions, it is possible for more counts to be generated by the MPU than are possible to transmit (i.e. the data link rate is smaller than the data production rate). In that case, the MPU will discard packaged events, which leads to an exposure gap. Events are packaged into 100-event groups, and the maximum transmission rate is about 55 packets per second, so shredding due to this cause will have tens of gaps per second. On the ground, screening can produce shredding effects as well. Some of the screening variables, such as FPM_OVERONLY_COUNT (overshoots) have statistical noise. If distribution of the variable is broader than the range specified for screening, then the result can be shredded GTIs. For example, if overshoots range between 1 and 20 ct/s, and the overonly_range is set to "0-10" then there will be many small GTIs as the overshoots vary back and forth over the "10" threshold. Shredded GTIs are not beneficial to scientific analysis. They make exposure calculation difficult and inaccurate In addition, the underlying causes (i.e. extreme count rates) would imply that instrument deadtimes are also extreme, again making exposure calculations potentially inaccurate. This release employs several new techniques to reduce the problems of Shredded GTIs. For noisy screening, now will filter out small GTIs due to noisy variables (overshoots and undershoots). For MPU shredding due to extreme count rates, niautoscreen is now more robust about letting some (or all) MPUs from skewing the allowed range. For the noisy screening portion of the issue, nicerl2/nimaketime have two new parameters called mingti and erodedilate which will remove shredded GTIs with "shred" size ~5 seconds or smaller. This is enabled by default.
In addition, the autoscreening tool, niautoscreen, enforces some
sanity checks, which prevent many MPUs with shredding from skewing the
screening. Previously, users sometimes reported that after
autoscreening, only about 8 detectors (one MPU's worth) were left
selected, and this is due to that skewing. The new algorithm enforces
stronger sanity constraints. However, a caveat to this enhanced
treatment is that users may find no good time after running
niautoscreen rather than ~8 detectors. This is the desired behavior
to prevent heavily "shredded" MPUs from biasing your spectral analysis.
HEASoft 6.32 / NICERDAS 11 does not involve published calibration changes. The existing published NICER calibration remains in effect. NICER analysis should continue to use NICER CALDB release xti20221001, which was released in November, 2022. Please see the NICER's Calibration Recommendations for current best practice, and NICER Calibration Documents for calibration release notes. For example, the required auxiliary files for background generation, and bad times for automatic screening, are included in CALDB. Also, this release has a new energy scale for both the slow (PI) and fast (PI_FAST) channels. The updates improve the cross-alignment of the energy scale between detectors, especially at high energies above 8 keV. This change has negligible effect below 8 keV (< 5 eV on average). Above 8 keV there are large effects, which increase to a maximum of ~300 eV for some detectors at 13 keV. In practice, few scientists will notice a change, since most NICER science counts are in the 0.3 - 6 keV range. Complete Change Log
Related TopicsFor instructions on setting up a NICER analysis environment, please see the Setting Up a NICER Analysis Environment thread. For more information on recommended settings and calibration recommendations, please see the NICER Calibration Recommendations analysis thread. For release notes for the previous release (HEASoft 6.31 / NICERDAS 10), please see the Software Release HEASoft 6.31 thread. Modifications
|
- HEASARC Director: Dr. Andy Ptak
- NASA Official: Dr. Andy Ptak
- Web Curator: J.D. Myers
- Last Modified: 23-Aug-2023