Software Release HEASoft 6.31.1 / NICERDAS 10a and Calibration xti20221001Return to: Analysis Threads | Analysis Main Page
OverviewThis page discusses new changes available in NICERDAS version 10, distributed in HEASoft 6.31 (and 6.31.1). This release is a major NICER software and calibration release, with signifigant improvements regarding background subtraction and overall workflow. Read this thread if you want to: Understand NICERDAS 10 software Last update: 2022-12-20 IntroductionObtaining useful scientific results from NICER observations is a combination of NICER observational data with software and calibration products. Over time, NICER is committed to improving software and calibration so that scientists can get the most out of their data. This page discusses changes available in NICERDAS 10, released with HEASoft 6.31 on November 15th, 2022, and a subsequent patch release 6.31.1 on December 19th, 2022 (see the 6.31.1 section below for more information on changes in the patch release). HEASoft 6.31.1 is a major NICER software feature release. There are significant background modeling and workflow improvements. There is also a new NICER calibration release, designated xti20221001, which was also relased in late October. It is recommded to upgrade to the latest patch release (HEASoft 6.31.1 / NICERDAS 10a). As always, we recommend users to document which version of software they are using, especially for publication. This will help future analysts repeat your results. For more information, see the Reporting NICER software and calibration versions thread. Overview of NICER ImprovementsHEASoft 6.31.1 (NICERDAS 10a) is a major NICER software release, and a corresponding NICER calibration update, xti20221001, is also released. This section provides an overview of the changes. More detail is provided in subsequent sections. The patch release HEASoft 6.31.1 (NICERDAS 10a) has the same featues as HEASoft 6.31, but additional bug fixes. Standard "Level 3" spectrum and light curve pipeline tools The tasks nicerl3-spect and nicerl3-lc provide a complete and standard spectral and light curve analysis chain. These tools perform all of the NICER-recommended steps to produce spectra (and associated backgrounds and responses) and light curves. The tools can also produce a handy graphical (.png) output. See the Details section below for more information. Background Modeling. This release includes three background models, integrated into HEASoft. It is no longer required to download background modeling software or auxiliary files separately since they are now included with NICERDAS. See the Details section below for more information. The background models included are:
Geomagnetic Data Now Required. Geomagnetic quantities are used by two of the three background models. These quantities include KP (interplanetary Kp index) as well as SOLAR_PHI (solar modulation potential). Now using geomagnetic data will be required. Users must download an updated set of geomagnetic data and set their GEOMAG_PATH environment variable to point to the location. To help users do this process, a new tool called nigeodown is provided which automatically downloads, extracts and installs geomagnetic data. Please see the Geomagnetic Quantities thread. Because multiple background models use geomagnetic data in support of their background estimates, specifying geomagnetic data is required and no longer optional. Please see the Setting Up a NICER Analysis Environment thread for more information about how to do this practically, as well as the Geomagnetic Quantities thread for additional details. By default, nicerl2 will automatically locate and use the geomagnetic data if it has be set up properly without additional command line settings. Automatic screening. The nicerl2 task now automatically screens user data for detectors that are noisy or misperforming. This screening is done on a pointing-by-pointing GTI. See the Details section below for more information. Default overshoot screening. The default overshoot screening criterion have been changed. There are two parameters to nicerl2 (and nimaketime) that control overshoot filtering: overonly_range and overonly_expr, and these were set to aggressively remove any type of background flaring. However, this also had the unwanted effect of sometimes removing all data from an observation, or creating a "shredded GTI" problem for borderline count rates. With background models now being included in NICERDAS, the overshoot screening can be relaxed. The new screening is overonly_range=0-30.0 (instead of 0-1.5, and with no overonly_expr). The nicerl2 task now automatically screens user data for detectors that are noisy or misperforming. This screening is done on a pointing-by-pointing GTI. See the Details section below for more information. Potential speed enhancements. The nicerl2 task can now be run with incremental=YES, which means that it will only reprocess existing data if necessary. See the Details section below for more information. Calibration Updates.NICERDAS also is meant to be paired with a new NICER CALDB release, designated xti20221001. In addition to the new CALDB files required to support the above capabilities, this release also has a new energy scale calibration which improves the high energy response (designated "optmv13"). See the Details section below for more information. Should I Reprocess My Data?Users will want to know, should they reprocess my data with this new release? The answer depends on the circumstances. Users new to NICER or starting a new project will likely benefit from reprocessing their data with the newest release. Also, the new background and high level product features in this release are compelling enough that users may want to reprocess their data to take advantage of the features. Users New to NICER. If you are just starting to use NICER data, the NICER team highly recommends that you begin your work with the newest NICER software and calibration. This release creates much more streamlined workflows for both spectral and light curve extraction, so it is probably not worth investing time learning older techniques. Starting a New Project. If you are starting a new project, the NICER team recommends that you update your software and calibration. Although the new energy scale released with this calibration update will provide marginal improvements in analysis, the new automatic screening tools may be beneficial to users. Users who have already developed their own workflow or have already extracted spectra or light curves, may not see a great benefit from going back to reprocess their data. However, the new tools and calibrations in this release will be the basis of future improvements, so moving analysis pipelines to the new release is definitely recommended when practical. Try Out New Background Models or High Level Product Pipelines. Since three background models are included with NICERDAS, this may provide users an opportunity to try these models out with their data without doing separate downloads. In addition, this release has two new high level ("Level 3") product pipelines, nicerl3-spect and nicerl3-lc, which perform all the recommended steps and make product extraction much simpler. Users may want to try out these tools, which require minimal learning curve and compare to their own workflow. A separate, but related, question is, am I required to reprocess my data to take advantage of these new features? The short answer is yes, you are required to reprocess your data with nicerl2 to use the new background modeling and high level product extraction tools. This is mostly because certain new filter file columns are required for the new features to work properly, but also because the old filter file generator (NICERDAS 8 and earlier) has some bugs that need repairing. Potential Incompatibility GotchasGenerally the NICER team strives to maintain forward and backward compatibility. This improves the user experience with our software and also instills confidence in the results from our observatory. However, in this release, there are the potential for some minor incompatibilities. Filter File Changes. The filter file (MKF file) columns have changed since the previous release. Some columns have been added and some removed (for more information, please see the Filter Files thread. Although it is advised to reprocess all of your data when starting a new project, there may be some situations where you have to merge filter files from across versions. Use the task nimkfmerge for this situation. Metadata keywords in NICER spectra. The NICER response generation tasks nicerarf and nicerrmf have the option updatepha. When updatepha=YES, keywords are written to the spectrum file that indicate to XSPEC which background and response files to use (BACKFILE, ANCRFILE, RESPFILE, etc). Unfortunately, this updatepha=YES feature did not work well in previous NICERDAS releases. Also if you included a path with subdirectories, these subdirectories were also included in the output keywords. These subdirectory names could make it difficult to use the file within XSPEC, especially if your working directory changes between the time that you generate the file and the time that you use the file within XSPEC.
If you did rely upon updatepha=YES working, please beware that
subdirectory components are now stripped away when written to the
output keywords. You can use the "cd" command within XSPEC to change
directories to the location of your data. However, the NICER team
recommends to set updatepha=NO everywhere, and explicitly load the
files that you need. The nicerl3-spect command will automatically
generate a "load" XSPEC script file that will help get you started.
NICERDAS 10 includes two new high level product pipeline tasks:
The new high-level product extraction tools nicerl3-spect and nicerl3-lc provide a straightforward way to obtain accurate spectral and light curve analysis results. The nicerl3-spect tool performs all of the NICER-recommended steps to extract a spectrum, generate responses (both ARF and RMF), and generate a background using the user's choice of background model. In addition, the task creates a basic XSPEC .xcm file (or pyXspec .py file) which can be used to load the data into XSPEC and diagnose it quickly. The nicerl3-lc tool extracts light curve for NICER data, and optionally performs background subtraction.
Both tools require that you have reprocessed your data with a current
version of nicerl2.
HEASoft 6.31.1 / NICERDAS 10a includes major advancements for NICER background modeling. Three background models are included in NICERDAS 10, including the well-known 3C50 and Space Weather models, and the new SCORPEON model. All three models are integrated within the high-level product tools. The high level product tools nicerl3-spect and nicerl3-lc allow the user to simply choose which model to use and the tools will automatically generate the products. In addition the user can run the tools separately if they wish. Existing 3C50 and Space Weather Models Now in NICERDAS. Before NICERDAS 10, two background models (3C50 and Space Weather) were available as a separate download from the NICER Background Estimator Tools page. These models (nibackgen3C50 and nispaceweather) are now included within NICERDAS and do not require separate downloads. All of the software and extra files required by these tools are now part of NICERDAS and NICER's CALDB release. Including these models should make using and trying background models easier for the user. SCORPEON. A new background model named SCORPEON is also included in this release SCORPEON taks a very different approach to background modeling. You can find more information about this model on the SCORPEON overview page. In SCORPEON, rather than generating a file which is meant to be subtracted from the spectrum within XSPEC, the SCORPEON approach is to model the background directly without ever having to subtract it. SCORPEON is composed of several components that are likely associated with real physical phenomena in the geomagnetosphere. The SCORPEON background model is a parameterized background model that can be fit in XPEC along with your source parameters. This has some major benefits. Done properly, you will never have background over- or under-subtraction problems, as you will likely have with other models. In addition, the uncertainties your source spectral parameters will properly reflect the covariance with uncertainties in the background, i.e. a more realistic error estimate. Also, by not subtracting background, you are able to use more appropriate statistics in the low counts regime such as Cash or "pgstat". If the user insists upon having a background file instead of XSPEC model, SCORPEON can generate this. The user will not get the full benefits of SCORPEON, but in some cases a background file is necessary. For more information on SCORPEON, please see the SCORPEON overview page. Existing Background Workflows. Please know that as these tools were integrated into NICERDAS, the existing interfaces for nibackgen3C50 and nibkgestimator were maintained. In other words, you can use the versions released with NICERDAS as drop-in replacements for the versions you downloaded separately. The new NICERDAS versions have some additional capabilities (such as to query CALDB for auxiliary files), but existing command line usages should be unchanged. Existing background workflows should continue to work, or you can migrate to the high level product pipelines. Therefore, if you have an workflow that uses the existing 3C50 (nibackgen3C50) or Space Weather (nibkgestimator) tools that you downloaded separately, you have several choices.
For all background models, the user is required to reprocess their
data with a current version of nicerl2.
NICERDAS 10 includes a powerful new automatic screening tool named niautoscreen. This tool checks each detector for problematic issues during each pointing good time interval (GTI). Any such detectors are excluded only during the pointings where the problems are not detected, thus giving the benefit of all good detectors when possible. The new niautoscreen task is run automatically by nicerl2 and will exclude detectors with problematic issues. The tool adjusts the event file's FPM Selection information. This allows the response generation tools to automatically construct an accurate response, even when some detectors are deselected for a portion of the time. The automatic screening tool checks various quantities that can indicate possible problems. The tool constructs robust quantities (median, robust standard deviation), which are used to determine which detectors are out-of-family, if any. The tool checks the following quantities on a per-detector and per-MPU basis:
niautoscreen is automatically run when you run nicerl2. You can
control its function by setting autoscreen=YES or autoscreen=NO.
The "Level 2" pipeline task nicerl2 has a new parameter incremental. When incremental=yes, the task will avoid reprocessing data when existing data is up to date. By setting incremental=YES when running nicerl2, significant time savings can be obtained, especially if the user is re-running nicerl2 multiple times with slightly different screening settings. nicerl2 automatically checks the metadata keywords for the correct information, and if the file is not up to date, reprocessing is forced. The steps that can be skipped when incremental=YES include re-calibration and filter file generation. Some steps are always done, regardless of the incremental setting. Please use this setting carefully. If you are upgrading your software or calibration, it is recommended to set incremental=NO to for a complete clean processing the first time, but it can then be set to YES for subsequent runs.
For more information about nicerl2, please see the
nicerl2 analysis thread.
Along with NICERDAS 10, the NICER team is also releasing a new version of CALDB. NICER CALDB release xti20221001 contains many new calibration updates required to support the new features of NICERDAS 10. Please see the NICER's Calibration Recommendations for current best practice, and NICER Calibration Documents for calibration release notes. For example, the required auxiliary files for background generation, and bad times for automatic screening, are included in CALDB. Also, this release has a new energy scale for both the slow (PI) and fast (PI_FAST) channels. The updates improve the cross-alignment of the energy scale between detectors, especially at high energies above 8 keV. This change has negligible effect below 8 keV (< 5 eV on average). Above 8 keV there are large effects, which increase to a maximum of ~300 eV for some detectors at 13 keV. In practice, few scientists will notice a change, since most NICER science counts are in the 0.3 - 6 keV range. Other UpdatesNICERDAS 10 contains several other significant updates. Filter file default column names have changed. The new "NICERV4" alias gives the default column names for this release. In general, some low-utility columns have been removed to save execution time and disk space, while some columns have been added. Please see the Filter Files thread for more information. The NICER team has discovered rare circumstances where an event file may contain events where the PI_FAST column is erroneous. This appears in the PI_FAST spectrum as a group of spectral line-like features around 3 keV. This problem has been traced to events that create an invalid pulse height in the fast chain, and a new tool (nicalfixevt) corrects this problem. This tool is automatically run when nicerl2 (or nicercal) is run. Complete Change Log
HEASoft 6.31.1 / NICERDAS 10a ChangesThe NICER team highly recommends upgrading to patch release HEASoft 6.31.1 (NICERDAS 10a), since it contains several important bug fixes. There are major reasons we strongly recommend NICER users install the patch release. These are due to bug reports or incorrect results being presented within XSPEC. The issues that are resolved are:
There are other more minor NICER-related changes to HEASoft. The issues noted here are not as critical as the ones above but may affect some scientists.
Related TopicsFor instructions on setting up a NICER analysis environment, please see the Setting Up a NICER Analysis Environment thread. For more information on recommended settings and calibration recommendations, please see the NICER Calibration Recommendations analysis thread. For release notes for the previous release (HEASoft 6.30 / NICERDAS 9), please see the Software Release HEASoft 6.30 thread. Modifications
|
- HEASARC Director: Dr. Andy Ptak
- NASA Official: Dr. Andy Ptak
- Web Curator: J.D. Myers
- Last Modified: 20-Dec-2022