Retrieving and Using Geomagnetic Quantities

Overview

Some NICER background models rely upon geomagnetic quantites such as "Kp" to operate. There is a recommended way to add these quantities to your filter file so that you can apply the background models.

Read this thread if you want to: Add geomagnetic information such as "Kp" to your filter file.

Last update: 2022-12-20

Introduction

NICER is a non-imaging instrument. It collects X-rays through concentrator optics to a silicon drift diode detector. You can think of this detector as a single-pixel device, or "light bucket" that captures every X-ray event that is concentrated through the optics.

In addition to X-rays, the detectors also collect background events. Many of these events are due to the space environment of NICER (which is the same as the space environment of the International Space Station, the host of NICER). NICER's orbit takes it through radiation belts, as well as the South Atlantic Anomaly. NICER's detectors may also capture solar or extra-solar cosmic rays. All of these backgrounds are typically considered nuisance events, which need to be modeled.

One of the current background models utilizes a "Space Weather" method to estimate the background. You can read more about the Space Weather model on the NICER Background Estimators page. A key input to the model is the state of the space weather environment. Although there are many ways to characterize space weather conditions, the NICER Space Weather model uses the Kp-index as its parameter.

Another model is the SCORPEON background model, which also uses geomagnetic data quantities. In this case, it uses the solar modulation potential, or SOLAR_PHI, in order to estimate the modulation of cosmic-ray related background, which may vary slowly over time.

NICER background models require geomagnetic data to function properly As of HEASoft 6.31, use of geomagnetic data are no long optional.

More Details of Geomagnetic Quantities

For the he planetary K-index, or Kp-index, is an estimate of fluctuation disturbances to the local magnetic fields. These disturbances are typically caused by the solar wind buffetting the earth's magnetosphere. These same disturbances may produce higher NICER background. The "p" in Kp means "planetary," as an effort to reconcile data from many earthbound magnetometer observatories into a single planet-wide quantity. For more information, consult the Wikipedia article on the subject.

There are two sources of compiled Kp-index data.

• NOAA. NOAA's Space Weather Prediction Center provides routine updates to Kp here. These compilations have a 3-hour sample cadence and resolution of 1 Kp-index point. NOAA provides a Kp forecast for a few days into the future.
• Potsdam. GFZ Helmholtz Centre Potsdam provides daily updates to Kp here. These compilations have a 3-hour sample cadence and a resolution of 1/3rd of a Kp-index point. Potsdam does not provide routine forecast data. Up to two months of Potsdam may be considered "quicklook" before definitive data is released. Thus, data up to two months old may be altered when definitive data become available.

The solar modulation potential data, or SOLAR_PHI, is compiled by the Oulu cosmic ray station, located in Finland. At the high geographic latitudes of this detector, cosmic rays are easily able to penetrate the earth's magnetosphere, and the neutron monitor at that station is a good proxy for total cosmic rays. Cosmic ray incident on the geomagnetosphere are not constant. The solar wind and magnetic field can also provide a partial barrier to lower energy cosmic rays. This barrier energy is known as the solar modulation potential, or SOLAR_PHI. The NICER team downloads and updates data from Oulu on a routine basis.

Geomagnetic Data Must be Updated Routinely

Geomagnetic data such as Kp and SOLAR_PHI vary from time-to-time and are not predictable. NICER's software uses these quantities to estimate the background levels, but they can only generate reliable and accurate estimates if geomagnetic data that covers the same time range as the observation are available. For this reason, the user must update their geomagnetic data routinely.

Typically, when starting a project, users should update their geomagnetic data. If users expect to be analyzing "realtime" or ToO data soon after they are collected, they should regularly keep their geomagnetic data up to date.

Geomagnetic Data Now Required by nicerl2 (Not Optional)

As of HEASoft 6.31 (NICERDAS 10), NICER software now requires you have updated geomagnetic data. Because the user can select one of several background models, and some of the models require different data, the choice was made to force users to have update geomagnetic data available. As described below, supported is provided to streamline this process.

Unlike previous software releases, HEASoft 6.31 and later require user to have up-to-date geomagnetic data.

The 'geomag_path' parameter of nicerl2 specifies where the task should find geomagnetic data. However, many users may find that storing this data in a location and using an environment variable GEOMAG_PATH, as described below, may be a simpler situation. This is in fact, the default.

Note that if you set geomag_path=NONE, you will disable geomagnetic-related columns in the filter file, which will prevent you from using most background models. This is also true if you set filtcolumns=NICERV3 or earlier (for compatibility), since background models do not have the proper information with older filter file column sets.

The GEOMAG_PATH Variable

By default, the nicerl2 task will automatically look for geomagnetic data in the directory indicated by the GEOMAG_PATH environment variable. This way they can set the GEOMAG_PATH once in their login files (.bashrc, .login, etc), and then not think about it again.

Thus, if the geomagnetic data are stored in /path/to/geomag the user would put the following in their login files.
.bashrc (bash or sh) export GEOMAG_PATH=/path/to/geomag
.login (csh or tcsh) setenv GEOMAG_PATH /path/to/geomag

If the geomag_path parameter is set to its default, nicerl2 (HEASoft 6.31 and later) will automatically find the geomagnetic data in this way.

Easy Way to Download Geomagnetic Data

NASA's HEASARC service compiles and provides the geomagnetic data required by NICERDAS.

As of HEASoft 6.31, NICERDAS has the tool nigeodown which will automatically download geomagnetic data, unarchive the data and install it. By default, it uses the GEOMAG_PATH environment variable as the location where geomagnetic files are installed.

Please note that you will need an active internet connection when you attempt to download geomagnetic data.

To run nigeodown, just type nigeodown and data will automatically be downloaded to the $GEOMAG_PATH directory.

To diagnose errors, increase the chatter level nigeodown chatter=3

To manually specify where geomagnetic data are downloaded to, nigeodown /path/to/geomag

Please be aware that nigeodown overwrites existing geomagnetic files with new ones. It ensures that data have been downloaded successfully before installing new data files on top of old files.

Installing Geomagnetic Data By Cron

The user may be tempted to user a cron job to download and install geomagnetic data on an automated schedule.

The NICER team does not recommend downloading geomagnetic data by automatic methods such as "cron."

There are several reasons for not recommending cron. First, downloading by cron means that the user's computer must always be awake and connected to the internet when the cron job executes. For many laptop users, this may not be the case.

Second, if the download fails for any reason, the geomagnetic files may be left in an incoherent state.

And finally, downloading new data while nicerl2 data processing is occuring could leave the resulting data in a questionable or partial state.

For these reasons, downloading by cron is an advanced usage that the NICER team cannot provide support for.

If the users insists on using cron, a cron entry to download data nightly at 02:00 will look something like this, 0 2 * * * export HEADAS=/path/to/headas; . $HEADAS/headas-init.sh; export GEOMAG_PATH=/path/to/geomag; nigeodown Please note that users may have to tune or tweak such an entry, and the NICER team cannot provide support for this.

Details of Geomagnetic Data Archiving

NASA's HEASARC service compiles these files for use with NICER and other missions. The data are retreived from their sources, formatted into FITS files, and stored here:

• https://heasarc.gsfc.nasa.gov/FTP/caldb/data/gen/pcf/geomag/

The mission-generic tool that can perform interpolation of the raw data into your tables is called geomagterp, which was released with HEASoft 6.27. However, the NICER tasks nicerl2 and niprefilter2 can make this effort simpler.

In order to use any of these tasks, you must

• Retrieve the geomagnetic data (two different methods described below)
• Apply interpolation using nicerl2.

Older HEASoft Versions: Retrieving Geomagnetic Data

The following information is provided for legacy installations (HEASoft 6.30 and earlier). If you have a recent HEASoft installation, please use nigeodown as described above.

The geomagnetic data are stored on HEASARC's CALDB area at https://heasarc.gsfc.nasa.gov/FTP/caldb/data/gen/pcf/geomag/. There are two ways you can retrieve the data for usage.

It will be important to know your geomag_path, which is the path where the files are stored. Each method described below tells you what that path is.

Obsolete Method 1: Download the data once. You retrieve a single tar file from HEASARC, and unarchive it. This is most beneficial if you are going off-line, or if you are going to process many files at once and you don't care about getting the newest Kp file while you are processing. Steps:

1. Download the tar file.
   Example obsolete code using curl:
   curl -O https://heasarc.gsfc.nasa.gov/FTP/caldb/data/gen/pcf/geomag/geomag.tar.gz Example obsolete using wget:
   wget https://heasarc.gsfc.nasa.gov/FTP/caldb/data/gen/pcf/geomag/geomag.tar.gz
2. Untar tar file. tar xvpzf geomag.tar.gz
3. Make a note of in which directory you place the results. This directory is your geomag_path. For example, if you executed the tar command in /path/to/geomag_data, then your geomag_path is /path/to/geomag_data.

Obsolete Method 2: Access the geomagnetic files over the network as needed. With this method, you access the files directly over the network each time they are needed. This is great if you are on the network, because you don't need to download or configure anything. The process happens automatically and you will always get the newest file when you use the correct geomag_path. A downside to this method is that each time the file is needed, it will be re-downloaded: large processing runs will be slowed down, and you will use network data repeatedly. So this method is not recommended if you are off-line, or if you are processing large amounts of data. Steps:

1. Your obsolete geomag_path is:
   https://heasarc.gsfc.nasa.gov/FTP/caldb/data/gen/pcf/geomag/

Obsolete Method 3: Combine both methods using a "cron" job. Use a cron job to automatically download geomagnetic data periodically, such as once per day, using Method 1. Although "cron" is not something we can help you with, there are many online resources that can.

In this step we will apply the geomagnetic data to a single observation. (HEASoft 6.30 and earlier). nicerl2 indir=1234567890 \ # (OBSOLETE - HEASoft 6.30 and earlier) geomag_path=/your/geomag_path \ geomag_columns="kp_noaa.fits(KP)" \ clobber=YES ...

where

• indir=1234567890 is your observation directory name
• geomag_path is the path to your geomagnetic data as described above.
• geomag_columns="kp_noaa.fits(KP)" will retrieve NOAA's Kp and save it as a column named KP
• to retrieve Potsdam's Kp-index instead, use
  geomag_columns="kp_potsdam.fits(KP)"
• if you want to give the column a different name, change the name enclosed in parentheses
• clobber=YES means that your data could be overwritten
• the "..." indicates you may want to use other options such as niprefilter2_coltypes

Note that "nicerl2" will completely reprocess your data from the ground up, applying all recommended calibrations and screenings. When it is complete, you will have a new filter file located in 1234567890/auxil/ni1234567890.mkf which has a column named KP (or whatever column name you requested in parentheses).

Older HEASoft Versions: Using "niprefilter2" instead

The following information is provided for legacy installations (HEASoft 6.30 and earlier). If you have a recent HEASoft installation, please use nigeodown as described above.

Although "nicerl2" is typically the most simple method to retrieve geomagnetic columns, it does reprocess all of your data from scratch. This can be rather time consuming if you simply want a new filter file. You can generate only a new filter file using "niprefilter2". Here is how to run only niprefilter2. Note that this technique is not recommended if you do need to apply new calibrations or new screenings to your data, since you might as well use the process above with "nicerl2" instead.

First, make sure your filter file is unzipped. As retrieved from the archive, the filter file is compressed using gzip (a .mkf.gz file). niprefilter2 will not be able to modify a gzipped file, so uncompress it first. gunzip 1234567890/auxil/ni1234567890.mkf.gz where the named file should be the path to your actual filter file. Now run niprefilter2 using the following: niprefilter2 indir=1234567890 \ infile=1234567890/auxil/ni1234567890.mkf \ outfile=INFILE \ clobber=YES \ geomag_path=/your/geomag_path \ geomag_columns="kp_noaa.fits(KP)"

where

• indir=1234567890 is your observation directory name
• infile=1234567890/auxil/ni1234567890.mkf is your filter file name
• outfile=INFILE and clobber=YES means to overwrite the existing filter file
• geomag_path is the path to your geomagnetic data as described above.
• geomag_columns="kp_noaa.fits(KP)" will retrieve NOAA's Kp and save it as a column named KP.

After running to completion, your old filter file will be replaced by a new one. If you do not want to overwrite your filter file, then set outfile to a differently named file.

Summary

Following the steps described above will add a column to your filter file with the Kp-index data, which will enable you to use certain background filtering models.

Modifications

• 2020-07-28 - further minor corrections
• 2020-05-27 - minor edits
• 2020-04-24 - initial draft
• 2021-04-16 - add navigation bar
• 2021-11-16 - correct tar command
• 2022-10-20 - update for HEASoft 6.31 (NICERDAS 10) and deprecate older usage
• 2022-12-20 - further editing to emphasize some sections are obsolete