Combining Multiple NICER Observations

Overview

It is a common occurrence that multiple NICER observations cover a single scientifically interesting time window. This may occur either because NICER observation of more than one day are naturally broken into segments of one day or less, but it could also occur if the analyst wishes to combine multiple visits into a single data set. This thread gives advice in how to do this.

Read this thread if you want to: Combine multiple NICER observations into a single data set.

Last update: 2024-01-16

Introduction

It is a common occurrence that multiple NICER observations cover a single scientifically interesting time window. There are several reasons for this to be true.

During the planning and pipeline processing steps, NICER observations are nevel allowed to be longer than a single UTC calendar day (starting at UTC midnight). If a science visit spans more than one calendar day, then the NICER visit can be broken into multiple segments. An analyst wishing to consider these data as a single object must somehow combine multiple segments into a single data set.

It is also possible that the analyst is interested in combining multiple visits of the same target into a single data set that spans it all. Here again the idea is the same: to take multiple NICER segments (a.k.a. sequences, a.k.a. OBS_IDs) into something more easily processed.

This thread gives advice in how to do this operation. There are actually multiple ways to achieve this goal, and we will discuss them, along with the pro's and con's of each.

• Method 1: Keep data sets separate. Procedurally, this method is the easiest because it requires no additional work. However the result of analysis is multiple spectra or light curves that must somehow be considered later.
• Method 2: Merge filter file and event lists. This method is the most sophisticated in the sense that the result is a single primary data set of a filter file and event list, from which "master" output products can be created.

Prerequisites

Here is what is needed:

• More than one NICER observation data sets.

Each data set is labeled by its observation ID, or OBS_ID. As retrieved from the HEASARC archive, an observation is stored in a directory with a ten-digit numerical name, and that name is the OBS_ID. For the purposes of the examples below, we will assume that the observations are in the current directory under the following names: ./1234567801 ./1234567802 ./1234567803 These example directory names are in a sequence (01, 02, 03) for this example, but this is not necessary, and in principle you can combine any NICER sequences that you wish.

We also assume that the directories are laid out as retrieved from the HEASARC NICER archive. The steps outlined below will not work if you have rearranged the directory layout.

Method 1: Keep data sets separate

This method is actually not a method at all. NICER data sets come pre-divided into segments. These segments are no larger than a UTC calendar day (days are broken at UTC midnight, with some tolerance so that observations that span midnight are not split into two). Observation segments may be processed using all of NICER's software tools to generate spectra and light curves unique to each of those segments.

Using Method 1 is easiest in the sense that one simply runs the standard tools for each observation, and gets one spectrum or lightcurve for each observation, but may result in many output products for long NICER observations.

When to use: Use this method for quick-look, or if NICER's division into daily segments meets the analyst's scientific needs. For bright or highly variable sources, it is likely that you will want to make spectral or light curves for daily or shorter time periods any way, in order to capture the variability of the X-ray target.

What to do: No up-front effort is required to use this method. However, any products you generate will be naturally split by observation segment. For each data set separately, run nicerl2, followed by nicerl3-spect and/or nicerl3-lc, respectively, in order to produce spectra or light curves.

How to use products: There are a couple of ways to use the resulting products, as detailed below.

Using Method 1 Products

You have a couple of options when using the products from this process.

• Analyze products separately. No special actions are needed to use this option, other than standard tools as noted above.
• Combining spectra is not recommended. If you are using this method, it is not recommended to merge spectra from different observations. Doing this exercise properly is more difficult than it may seem at first glance.
• Loading multiple spectra into xspec. Instead of merging multiple spectra, we do recommend that you load multiple spectra into xspec. Doing so is beyond the scope of this thread, but the basic concept is to use multiple 'data' commands to xspec, such as this: data 1:1 ./1234567801.pha data 2:2 ./1234567802.pha data 3:3 ./1234567803.pha These XSPEC commands will load three independent spectra. Of course you can change the commands to suit your needs, and add more data commands if you need to. You may also need to load responses matrix information as well. The benefit of this approach is that you gain the statistical advantage of using all of your data with a single spectral model.

  If you are using the SCORPEON model, see the Using SCORPEON with XSPEC page for more details on how to load multiple spectra.

• Combine light curves using lcurve. If you are trying to combine light curves, you can use the task named 'lcurve'.
  1. Make "list file" of light curves. You need to create a text file containing a list of light curve file names, commonly described as a 'lis' file. In this example, let us assume that you have created a light curve for each observation named ./1234567801.lc, ./1234567802.lc, etc. Create a text file containing these file names. Here is a sample file according to our three observations, which we will name 'lcurve.lis': ./1234567801.lc ./1234567802.lc ./1234567803.lc You can edit this file your self, or use the 'ls' command with wildcards. For example, ls ./12345678*.lc > lcurve.lis will create 'lcurve.lis' with all of the .lc files that match the pattern. Of course you should change the file (or pattern) to match your situation.
  2. Run lcurve. You can run it iteractively, or by the command line. If you run it interactively, you should run the command lcurve by itself and then follow the prompts: Number of time series for this task: 1 Ser. 1 filename: @lcurve.lis Name of the window file: - Newbin Time or negative rebinning: -1 Number of Newbins/Interval: INDEF Name of output file: master.flc This set of inputs will define "1" input series, whose filenames will be taken from lcurve.lis, with no window file, no rebinning and default number of time bins. The output will be saved to master.flc. You can achieve the same thing from the command line using the following command. lcurve 1 @lcurve.lis window=- dtnb=-1 nbint=INDEF outfile=master.flc Note that lcurve will offer to plot the light curve by default.
  3. The new light curve in this example is master.flc, but of course you can chose a different name. The resulting file will have "null" values for missing bins.
• Combine light curves using ftmerge. In this case we will use the utility tasks ftmerge and ftsort to make a sorted, merged light curve file.
  1. Make "list file" of light curves. You need to create a text file containing a list of light curve file names, commonly described as a 'lis' file. In this example, let us assume that you have created a light curve for each observation named ./1234567801.lc, ./1234567802.lc, etc. Create a text file containing these file names. Here is a sample file according to our three observations, which we will name 'lcurve.lis': ./1234567801.lc ./1234567802.lc ./1234567803.lc You can edit this file your self, or use the 'ls' command with wildcards. For example, ls ./12345678*.lc > lcurve.lis will create 'lcurve.lis' with all of the .lc files that match the pattern. Of course you should change the file (or pattern) to match your situation.
  2. Use ftmerge and ftsort to create a merged light curve. The following brief command: ftmerge @lcurve.lis - | ftsort - master.lc TIME will first merge the light curve file whose names are listed in lcurve.lis, and then sort them by the TIME column, putting the result in master.lc. Unlike lcurve, ftmerge does not do anything special to the light curve bins to align them or account for gaps. Gaps are only signaled by missing time samples.

Method 2: Merge filter file and event lists.

This method involves combining all of the desired NICER data associated with a particular target into a single merged data set. The user can then extract products using the standard product tools.

Using Method 2 provides the most flexibility, but may result in very large merged data sets if count rates are high.

When to use: Use this method for combining long observations, to look for faint sources by increasing exposure, or to make long-term light curves.

When not to use: Combining many data sets into a single merged dataset may result in huge file sizes, which may be cumbersome to use, consume considerable disk space, and take long times to process. The user must be aware that long observations may span different periods of X-ray target behavior as well as different background conditions. Care must be exercised when filtering a merged product to take these into account.

What to do: Use the tool niobsmerge to combine data sets. This tool combines the filter file, screened event lists, and (optionally) the orbit file. The inputs of this tool are:

• A list of observation directories.
• An optional output directory name (by default products are placed in the current working directory).
• An optional desired new "OBS_ID" ot use for the merged data sets.

To construct an observation list, use the "ls" command and redirect the results to a file. Here is a command which lists the observations we have outlined in the example above: ls -d ./12345678?? > ./obsids.lis This command uses the wildcard pattern ./12345678?? to select all observation datasets that begin with 12345678 and end with another two-characters, which matches NICER's 10-digit obsid names. The "-d" flag to ls prevents recursive file listing of directories. The results are directed to the file named obsids.lis in the current working directory.

We will place the outputs in a new directory named 1234567800. We pick this name since all observations start with 12345678, and the "00" can indicate that this is a "merged" data set. NICER data sets never end with a trailing "00", so using "00" for the last two digits is a unique and helpful indicator of a merged data set.

We run niobsmerge like this, niobsmerge @obsids.lis outdir=./1234567800 obsid=1234567800 where

• obsids.lis is the list of observation data sets as described above
• outdir is the output directory name as described above
• obsid is a new "OBS_ID" to use for this data set; here we have chosen 1234567800, with the trailing 00, again to indicate a merged data set. Any obsid string is allowed however, including both letters and numbers. If you don't specify an obsid, the OBS_ID of the first listed data set will be used.

There are many other options to niobsmerge. Please see the niobsmerge help page for more information.

The result of niobsmerge is a fully merged set of event files (both UFA and CL), a merged MKF file, and (optionally) a merged orbit file. These files are placed in the output directory, and are named using standard naming for NICER files, such as niNNNNNNNNNN_0mpu8_ufa.evt, where NNNNNNNNNN is the obsid specified on the command line.

Note: if you are not using the 3C50 model, you do not need to merge the UFA event files. Not merging the UFA files save considerable disk space and execution time. To disable merging the UFA files, set ufafile=NONE on the command line.

How to use products: At this stage, you have a combined data set. You can run the standard tools nicerl3-spect for spectral analysis and nicerl3-lc for light curve analysis. Examples: nicerl3-spect indir=./1234567800 clobber=YES nicerl3-lc indir=./1234567800 clobber=YES where "indir" is the name of the directory containing merged products.

After running either of these product scripts, any downstream tools, such as XSPEC or lcurve, can be used as normal.

Other Methods Such as 'addspec' or 'addascaspec'

The NICER team does not recommend using spectrum adding tools such as "addspec" or "addascaspec" to combine spectral data sets. While it may be tempting, there are a number of problems.

First of all, these tools have not been tested with NICER data and can't be verified to produce valid results.

Second of all, if you are using a background model such as SCORPEON, there is no way to combine the "model" files using any of those tools. You will not have a valid background estimate.

Modifications

• 2023-07-11 - initial draft
• 2024-01-16 - fixed broken link to scorpeon-xspec