HEASARC Staff Scientist Position - Applications are now being accepted for a Staff Scientist with significant experience and interest in the technical aspects of astrophysics research, to work in the High Energy Astrophysics Science Archive Research Center (HEASARC) at NASA Goddard Space Flight Center (GSFC) in Greenbelt, MD. Refer to the AAS Job register for full details.

Search:

[Advanced Search]



Software Release HEASoft 6.34 / NICERDAS 13

Overview

This page discusses new changes available in NICERDAS version 13, distributed in HEASoft 6.34. This release is a minor NICER software release, with one significant bug fix and other usability enhancements. Upgrading is recommended. There are no calibration changes in this release.

Read this thread if you want to: Understand NICERDAS 13 software

Last update: 2024-09-25

Introduction

Obtaining useful scientific results from NICER observations is a combination of NICER observational data with software and calibration products. Over time, NICER is committed to improving software and calibration so that scientists can get the most out of their data.

This page discusses changes available in NICERDAS 13, released with HEASoft 6.34 in August 2024.

Compared to HEASoft 6.33, this release is an minor NICER software feature release. However, there is one significant bug fix related to how the recommended workflow functions with data taken during the optical light leak. Because of this bug fix, we recommend that users upgrade their HEASoft installation and re-run nicerl2 completely on any datasets where they did separate day / night processing.

As always, we recommend users to document which version of software they are using, especially for publication. This will help future analysts repeat your results. For more information, see the Reporting NICER software and calibration versions thread.

Overview of NICER Improvements

HEASoft 6.34 (NICERDAS 13) is a minor NICER software release. This section provides an overview of the changes. More detail is provided in subsequent sections.

Bug fix to downgraded filter (mkf) file when using recommended nicerl2 screening for orbit day. NICER's recommended approach for dealing with the optical light leak is to run nicerl2 twice, once with the default (orbit night) and once again with tasks=SCREEN to pick up orbit day data. If you have BOTH a gzipped and non-gzipped mkf file, the second run would use the wrong version. This will lead to incorrect results. HEASoft 6.34 fixes this bug. See the MKF Bug Fix section below for more information.

Other Usability Improvements. There are a number of usability improvements that do not affect the quality of science output but make the NICER workflow a bit more streamlined. Please see the Usability Improvements section below for more information.

Background Model Tracability. Beginning with this release NICER's background models will add keywords to the background files that indicate metadata about the model such as version information. This will aid in tracability of scientific results. Please see the Tracability section for more information.

Calibration Updates? There are no new calibration changes associated with this release. The recommended calibration release continues to be xti20240206. Please see NICER Calibration Recommendations for more information. Users can also refer to the Setting Up a NICER Analysis Environment page for more details on updating their calibration release.

Users should maintain their NICER calibration at xti20240206.

Should I Reprocess My Data?

Users will want to know, should they reprocess my data with this new release? The answer depends on the circumstances.

Users new to NICER or starting a new project will likely benefit from reprocessing their data with the newest release.

Impact of nicerl2 bug. If users have used the recommended workflow for optical light leak data (running nicerl2 once with tasks=ALL and once with tasks=SCREEN), they may find that their filter file (MKF file) is downgraded to the archive version. If you have used this workflow using HEASoft 6.33 or earlier, it is recommended to reprocess your data to regain an up-to-date filter file.

Users New to NICER. If you are just starting to use NICER data, the NICER team highly recommends that you begin your work with the newest NICER software and calibration. This release creates much more streamlined workflows for both spectral and light curve extraction, so it is probably not worth investing time learning older techniques.

Starting a New Project. If you are starting a new project, the NICER team recommends that you update your software and calibration.

A separate, but related, question is, am I required to reprocess my data to take advantage of these new features? The short answer is no, because there are no changes to the filter file or event calibration in this release.

Potential Incompatibility Gotchas

Generally the NICER team strives to maintain forward and backward compatibility. This improves the user experience with our software and also instills confidence in the results from our observatory. However, in this release, there are the potential for some minor incompatibilities.

Filter File Changes. There are no changes to the filter file columns in this NICER release.

nicerl2 Bug Fix. Users may find that their filter file has been downgraded without their explicit knowledge, if they used nicerl2 with tasks=SCREEN (recommended for optical light leak data). After the bug fix, it is best practice to re-run nicerl2 using tasks=ALL to upgrade their filter file. See the MKF Bug Fix section below for more information.

Background file changes. There are no quantitative changes to the NICER background models in this release. However, users may notice some small changes related to metadata keywords. Please see the Tracability section for more information.

New screening by default. There are is new screening in this release.

Details: nicerl2 MKF Bug Fix

As of May 2023, NICER has experienced damage to its optical blocking films which has led to increased levels of optical loading on its detectors. To mitigate this, the NICER team has taken several steps to reduce the impacts. One of these changes it to alter the low energy threshold during NICER's orbit day period. For more information on this topic please see the optical light leak pages.

In short, NICER's threshold is the detector lower cutoff energy. X-ray photons below this energy are not detected (although we discuss complications to this statement shortly). Thus, changes to the threshold change the effective NICER energy range.

Not all of NICER's software is able to handle this threshold change, so the NICER team recommends to run nicerl2 twice, once to collect data observed under orbit night conditions, and then again to collect data observed under orbit day conditions. Keeping these datasets separate allows the software to work.

To save run-time, the NICER team promotes a special way to invoke nicerl2 using the option tasks=SCREEN. Once all the compute-intensive work is done (tasks=CALMERGE,MKF), the actual screening can be run relatively quickly. For more information on this time-saving technique, please see the nicerl2 Pipeline analysis thresad.

However, in HEASoft 6.33 and earlier, there was a bug in the nicerl2 task which caused problematic behavior for the MKF file. If

  • BOTH a niNNNNNNNNNN.mkf and niNNNNNNNNNN.mkf.gz file exist in the auxil/ directory AND
  • nicerl2 is run with tasks=SCREEN, THEN
nicerl2 will remove the niNNNNNNNNNN.mkf file and unzip the niNNNNNNNNNN.mkf.gz file to replace it. The danger here is that the gzipped version probably came from archive and is therefore "old." The original mkf file is removed with little warning.

In summary, the bug will cause a user's existing filter file to be overwritten by an old file, when running nicerl2 with tasks=SCREEN. It is difficult to detect that this situation has occurred.

The impact of this bug is that some columns needed by downstream software could be missing from the "old" gzipped file, and/or filter file processing improvements could be lost. This could, for example, lead to incorrect background estimates or error messages.

The remedy for this situation is to upgrade to HEASoft 6.34 and re-run nicerl2 with task=ALL. The filter file will be recreated with the most up-to-date algorithms. The new version of nicerl2 will avoid re-using the gzipped file if tasks=SCREEN.

If you are unsure if your data is affected by this problem, we recommend to reprocess your data using nicerl2 and tasks=ALL before doing any further data extraction.

Details: NICERDAS Usability Improvements

HEASoft 6.34 has a number of small usability improvements. This section describes many of them.

nicerl2 checks for writable output directories. This prevents cryptic error messages.

nicerl2 is more flexible about input files. You can now gzip your calibrated and screened event ("UFA") file from a previous run and nicerl2 will find it. It also does not require the unscreened event file ("UF") to be present if tasks=SCREEN.

nicerl3-spect and nicerl3-lc handle task arguments better. The 'bkgconfigs' parameter is now passed to the SCORPEON background generator. For light curves, the 'lcthresh' parameter is passed to niextlc.

A new surface brightness profile 'disk' has been added to the response generators. When running nicerarf, set profile=disk and profpar=radius:value where value is the radiuso of the disk in arcseconds.

For the SCORPEON model, the XSPEC range of SAA norm ('saanorm') is now expanded to a more useful range. This does not affect the spectral shape of the SAA background model or how the norm is computed, but it does change the allowed range in XSPEC to 0-6000. It is rare that users analyze data within the SAA boundaries, so this will likely not impact any users.

Details: Background Model Tracability

Keeping track of and reporting which analysis was performed on NICER data is an important aspect of scientific rigor. For the purposes of repeatibility, the scientist should track what has been done to the data.

To aid in this effort, the NICER tools now provide more documentary keywords within output products which make NICER background analysis more tracable.

The standard NICER pipeline tools now append certain metadata keywords which survive to the final spectral and light curve products. No special settings are required: these keywords are automatically included.

Here is a list of the new keywords:

  • NIBKGTYP. The type of background model being used ('SW' = Space Weather, '3C50', 'SCORPEON').
  • SCORPVER. The SCORPEON model version number, eg. 'v23'.
  • SCORPVAR. The SCORPEON model variant used, eg. 'nxb-detailed'
  • SCORPCOM. The SCORPEON model background components included, e.g. 'sky,nxb'.
  • BGYR3C50. The 3C50 model's version string, equivalent to 'gain year'.
  • SWVERSN. The Space Weather version string.

To determine the values of these keywords, the user can use the ftlist command to print the keywords. 

  ftlist niNNNNNNNNNN_sr.pha K
and search for the according keywords.

In addition to background information, the user should also document other items such as software and calibration versions. For more information, see the Reporting NICER software and calibration versions thread.

Complete Change Log

  • nicerl2:
    • checks for writability of output directory (cldir)
    • checks for gzipped version of ufa file as a fallback
    • does not require per-MPU "uf" files to be present if tasks=SCREEN
    • if using tasks=SCREEN, doesn't overwrite .mkf file with .mkf.gz file
    • better aggregation of filtcolumns entries so that it is possible to use -colname to remove desired columns
  • nicerl3-spect and nicerl3-lc:
    • bkgconfigs now passed to SCORPEON backgroun estimators
    • nicerl3-lc passes lcthresh parameter to niextlc
    • nicerl3-lc default value of noticerange now correctly reflects channel range of NICER
  • nivignette:
    • new "profile=disk" surface brightness profile
    • bug fix to prevent error when reading surface brightness profile FITS file
    • better error checking when radial profile is provided
  • all background estimation tools (SCORPEON, Space Weather, 3C50):
    • write metadata keywords to document the background model and configuration used
    • NIBKGTYP is the background model type (SW,3C50,SCORPEON)
    • SWVERSN is Space Weather version
    • BGYR3C50 is 3C50's "gain year" used
    • SCORPVER is SCORPEON's bkgver
    • SCORPVAR is SCORPEON's bkgvariant
    • SCORPCOM is SCORPEON's bkgcomponents
    • corresponding code in nicerl3-{lc,spect} to transfer metadata to product files
  • niscorpspect and niscorpspectmod:
    • range of saa_norm is now 0-6000, which is a better representative of the true range; this only affects people trying to obtain data taken during SAA
  • nicerarf:
    • support for nivignette's "disk" surface brightness profile (also nicerl3-spect)
  • niobsmerge:
    • make sure that OBS_ID is written as a string and not an integer

Related Topics

For instructions on setting up a NICER analysis environment, please see the Setting Up a NICER Analysis Environment thread.

For more information on recommended settings and calibration recommendations, please see the NICER Calibration Recommendations analysis thread.

For release notes for the previous release (HEASoft 6.33 / NICERDAS 12 / 12a), please see the Software Release HEASoft 6.33 thread.

Modifications

  • 2024-09-25 - initial draft