HEASARC Staff Scientist Position - Applications are now being accepted for a Staff Scientist with significant experience and interest in the technical aspects of astrophysics research, to work in the High Energy Astrophysics Science Archive Research Center (HEASARC) at NASA Goddard Space Flight Center (GSFC) in Greenbelt, MD. Refer to the AAS Job register for full details.

Search:

[Advanced Search]



Software Release HEASoft 6.29 / NICERDAS 8 and Calibration Release xti20210707

Overview

This page discusses new changes available in NICERDAS version 8, distributed in HEASoft 6.29, as well as corresponding NICER calibration changes released as xti20210707. This is a major new release of software and calibration products that enabled several new key capabilities.

  • Response calculators that generate ARF and RMF responses for your specific observational spectrum;
  • New calibration products to support the response calculators, as well as a new energy scale solution ("optmv12") that should allow scientists to work with more "good" data
  • A new system for selecting and deselecting detectors, and keeping track of enabled detectors, called "FPM Selection" information
  • Updated filter files and default screening criteria

Read this thread if you want to: Understand NICERDAS 8 software and calibration changes

Last update: 2021-07-20

Introduction

Obtaining useful scientific results from NICER observations is a combination of NICER observational data with software and calibration products. Over time, NICER is committed to improving software and calibration so that scientists can get the most out of their data.

This page discusses changes available in NICERDAS 8, released with HEASoft 6.29 in mid-July, 2021. To go along with the new software release, NICER is also releasing new calibration products which enable the software to work.

As always, we recommend users to document which version of software they are using, especially for publication. This will help future analysts repeat your results. For more information, see the Reporting NICER software and calibration versions thread.

In the following sections we will discuss the new changes to NICER software and calibration products.

Does This Mean My Old Analysis is Invalid?

Does this new software release invalidate previous analysis or published scientific results? The short answer is, NO, previous analysis remains valid to the extend that analysts have stayed within the Domain of Applicability published in the release notes. Please see NICER's Calibration Documentation page for release notes and other calibration documentation.

The new software enables new capabilities - primarily calculation of responses for off-axis and/or diffuse targets - and extends the existing calibration - primarily the energy scale solution to higher levels of optical loading.

Especially for on-axis targets, and for targets observed with the default screening criteria, there should be no pressing need to reanalyze or republish "old" NICER data. However, for new data or new analysis of old data, we do recommend upgrading to the most current software and calibration.

NICER Software Version Interactions

Can I use an older version of the software with the newer calibration or vice versa? The answer is mostly, "yes," but here are some details.

  • Newest software with newest calibration. (Recommended) It is recommended that you upgrade both HEASoft software and NICER CALDB at the same time. This gives you the best NICER-recommended tasks and calibration.
  • Older software with newer calibration. Using "older" HEASoft 6.28 (NICERDAS 7) or earlier with the newest calibration (xti20210707) will allow you to get the new "optmv12" gain solution, but many new software capabilities are not available.
  • Older calibration with newer software. Using HEASoft 6.29 (NICERDAS 8) with older calibration (xti20200722 or earlier) will produce valid results, but not let you take advantage of the "optmv12" gain solution. The response calculators will be installed but non-functional because of missing CALDB data.

In short, mixing old with new will not produce erroneous results, but you will not be able to take full advantage of the new software and calibration benefits.

NICER Software Changes: Response Calculators 'nicerarf' and 'nicerrmf'

NICER software now provides the tasks 'nicerarf' and 'nicerrmf' which compute the Ancillary Response File (ARF) and Redistribution Matrix File (RMF) responses for a particular conditions of your observation and spectrum.

The previous release did not provide specific software to do these tasks. Instead, the NICER team provided "canned" or precomputed responses and the user was responsible to combine these files into their desired response. The new tasks perform these steps for the analyst in a reliable way.

Why is this important? The NICER response is not fixed. Several conditions can change the response in a non-trivial way. In particular,

  • An off-axis target due to imperfect pointing or intentional offset pointing will have a different ARF due to vignetting.
  • A target observed in bright optical light conditions (near the sun or in certain orbit-day configurations) may have different spectral resolution than nominal.
The new software tasks handle these conditions appropriately.

For more information, please see the NICER Responses analysis thread.

NICER Software Change: FPM Selection Information

To generate an accurate response (both ARF and RMF), one has to know which of NICER's 52 detectors are enabled and selected for use. NICER now includes in event files additional information, called "FPM Selection" information that enables the response generation tasks to properly determine which detectors to include, and with what weight.

FPM Selection information is stored within the calibrated event files (the "ufa" and "cl" files). As long as the analyst uses NICER tools to manipulate event files, this information will be maintained and carried along. The ARF and RMF generators can automatically use this information.

A new software task, 'nifpmsel', is provided which can select or deselect detectors from an event file. This allows the analyst to select only certain detectors, or exclude certain known "bad" detectors from analysis. The "FPM Selection" information is updated so that an accurate response can be calculated.

Analysts who have developed their own workflow to filter or modify detectors from an event file may still do so. However, they should be aware that then the "FPM Selection" information in the event file may become stale or removed entirely, and it will not be possible to automatically apply FPM selection information during response generation. It is up to the analyst to pass detector selection information directly to the response generators.

The old technique of precomputed response matrices will remain in place for some time, but is now obsolete and all users should move to the new response generator tools.

For more information on FPM Selection information, please see the NICER FPM Selection Information analysis thread.

NICER Software Change: nicerl2 more fine-grained control

nicerl2 is a workhorse analysis script that does a large amount of standard processing for a given observation. nicerl2 performs calibration, generation of the MKF, and screening of the "ufa" event file to produce a "cl" cleaned event file.

However, many users run into simple performance issues where they want to change a small filtering / screening criterium, and that requires them to run the entire script.

nicerl2 now has a way to control more specifically what tasks are performed. For example, if the user just wants to re-screen their data, there is no need to recalibration or generate the MKF file again. This is accomplished with the new niprefilter parameter called 'tasks'.

For most users, once they have fully run nicerl2 once (with tasks=ALL), and wish to simply change a screening criterium, the can re-run nicerl2 with tasks=SCREEN.

For more information, please see the 'nicerl2' help file.

NICER Software Change: Default overshoot and undershoot criteria changed

NICER implements standard screening defaults for the processing pipeline, and for analysts to start with when analyzing data.

NICER now recommends to increase the following ranges, as specified as defaults to the 'nicerl2' and 'nimaketime' tasks. These changes should allow more data to survive the standard filtering, including most sun angles, with excellent calibration results.

  • overonly_range="*-1.5" increases the range by 50% (was *-1.0). This allows more data to be included in the standard screening and reduces the chances of noise in the overshoot rate from causing a "shredded" good time interval.
  • underonly_range="0-500" increases the range by 150% (as 0-200). This allows more data at higher optical loading conditions. With the new calibration energy scale release (see below), the NICER team has a valid energy scale to higher optical loadings. This should allow most observations to be analyzed with minimal data loss.
In addition, the overshoot expression screening criterium ('overonly_expr') automatically adjusts to match the maximum limit of the overonly_range. Thus, if you want to relax the over shoot limits, you need to only change the 'overonly_range' parameter and the overonly_expr will adjust automatically to match. (Of course you can also set the overonly_expr manually if you wish.)

NICER Software Change: Merging filter files made easier

Merging NICER filter files can be a challenge, especially if these files were generated by different NICER software versions. If any of the columns do not match exactly, the typical task 'ftmerge' will refuse to combine the files.

With this release, NICER introduces the task 'nimkfmerge' which can merge any filter files, even ones with mismatched columns. This will make it easier to make a full-observation filter file which stretches across many observation segments.

If columns exist in some files and not others, those columns will be filled with NULL values upon output.

NICER Software Change: Filter file additions

NICER provides a filter file (also known as an "MKF" file). This file is used by the default screening to produce a cleaned event list, but also contains many quantities that may be useful for diagnosing issues or generating background estimates.

In this release, the NICER software tasks niprefilter and niprefilter2 add more columns to the filter file that may be of use, or are used by NICER software to perform necessary computations.

For more information on changes to filter files, please see the Managing NICER Filter Files analysis thread.

NICER Software Change: CALDB accounting change

In previous versions of NICER software, the CALDBVER keyword was left unchanged, even if event files were recalibrated.

Beginning with this software release, the 'nicerl2' software task will automatically change the CALDBVER for affected files. This will allow greater transparency of which calibration version applies to which files.

NICER Calibration Update: Release xti20210707

In combination with this software release, there is a new CALDB release. The release is designated "xti20210707" and is recommended for all users.

The major changes to calibration in the xti20210707 release are,

  • New energy scale ("optmv12") which extends the energy scale solution up to undershoots as high as 500 undershoot counts per second per detector. This file is meant to be used in combination with the new software which changes the default underonly_range=0-500.
  • New files to support the response calculator tasks

For more information on recommended settings and calibration recommendations, please see the NICER Calibration Recommendations analysis thread.

For more information on the contents and performance of this release, please see the NICER Calibration Files page.

In particular,

  • NICER-xti20210707-Release-Notes - release notes for xti20210707
  • NICER-Response-History-20210707 - release history for NICER responses
  • NICER-Cal-Energy-Scale-optmv12-20210707 - release notes for NICER energy scales, including optmv12

Modifications

  • 2021-07-16 - initial draft