Search:

[Advanced Search]



Things to watch out for with XRISM data processing and analysis

Email the XRISM Science Data Center (SDC) helpdesk with any questions: XRISM-SDC-help@lists.nasa.gov

NOTE: The remote CalDB option fails for making ARFs (and running xrtraytrace) prior to HEASoft version 6.35. For earlier versions of the software, you can only use the local CalDB option to do XRISM spectral analysis.

General

  • GEN-1: Products
    • The files in the 'products' directory for each instrument are meant for a quicklook preview, and not for any detailed analysis. Spectra and responses should be produced for each source using the XRISM ftools.
    • The Xtend region file should correspond to a circle with radius 2.5' (85 pixels), but the radius is 3.83' (130 pixels), although the name of the file implies the former radius. You can edit the file to change the radius. Note that the circles in the Xtend images also have a radius of 3.83', not 2.5'.
    • There are some known cosmetic errors in the pipeline preview products. All of these errors will be fixed soon.
  • GEN-2: Response files
    • The canned responses (see Response files for proposers) are designed for simulations. Although they have the standard energy channels and can be loaded in Xspec with spectra extracted from flight data, the actual responses (RMF and ARF) are observation-dependent and should be generated for each application for proper spectral fitting.
  • GEN-3: xapipeline / rslpipeline / xtdpipeline
    • 'xapipeline' is the user pipeline tool that calls each instrument pipeline tool, 'rslpipeline' and 'xtdpipeline', to reprocess the data. Any of these tools can be run by hand. See the help for each of these tools for more information.
    • In general, the SDC recommends that users save reprocessed files or analysis results outside of the original archive data directory. Otherwise a FATAL error may result due to the presence of duplicate files within the parent (input) directory tree.
      • For example, use 'xapipeline indir=000145000 outdir=000145000_resolve_reproc', instead of 'outdir=000145000/resolve_reproc'
  • GEN-4: Region files for xaarfgen: Xtend region files input to xaarfgen can only be in RA/DEC or DET coordinates (selected by the 'regmode' parameter). Note that SKY coordinate region files are not the same as RA/DEC region files. Resolve region files can only be in DET coordinates.
  • GEN-5: 'erange' parameter in xaarfgen: The default value of the lower limit of 0.1 keV should not be used as it will result in an erroneous ARF. For GV closed, the value should not be less than 0.5 keV (0.3 keV for GV open). The function of the 'erange' parameter is not straightforward: the output ARF will not necessarily cover the range that is requested by the parameter because xaarfgen will choose internal energy grid points that are the nearest matches in a particular CalDB table to the values in 'erange'. If the output ARF does not cover the desired energy range, re-run xaarfgen with a wider energy range.

Resolve

  • RSL-1: Data format and naming conventions
    • The event files (and the spectra extracted from them) are split according to the filter wheel setting:
      • p0px0000 UNDEF
      • p0px1000 OPEN
      • p0px2000 Polyimide
      • p0px3000 ND
      • p0px4000 Be
      • p0px5000 Fe55
    • In general, you should use 'p0px1000' events for most science or calibration data analysis, unless the ND filter was used.
  • RSL-2: Energy scale assessment
    • Energy scale reports for all observations are available at https://heasarc.gsfc.nasa.gov/FTP/xrism/postlaunch/gainreports/. For any observation of interest, examination of the report for that OBSID is required in order to assure that the energy scale accuracy and spectral resolution are nominal and, therefore, standard Resolve data analysis techniques are applicable. If an energy scale report was not generated, it is likely that no (or insufficient) Fe55 calibration spectra for gain drift correction were acquired - as is the case for some early (check-out, commissioning) phase data and observations that experienced Filter Wheel Electronics anomalies. Each report includes a list of energy-report specific things to watch out for and caveats and to attend to. If there are gain drift correction solution failures users may need to omit data in a specific pixel or pixels, in a specific time interval, or in a specific pixel or pixels over specific time interval (see version 3.1 of the Quick-Start Guide for details). If there are missing gain drift correction solutions, the impact depends on how its absence affects the gain history.
    • There is a gain shift between Hp and Mp events, but one that is significantly mitigated by processing using HEASOFT 6.35 or later with XRISM CALDB 10 or later. Thus, reprocessing using xapipeline is required for archived observations processed with pipeline version earlier than 03.01.014.010 (2025 May 22). See version 3.1 of the Quick-Start Guide for details on handling of Mp events.
    • Pixel 27 exhibits, as yet unexplained, gain excursions. Due to the relative infrequency of gain fiducial measurements, the calculated gain drift correction may not capture these excursions, rendering the Pixel 27 energy scale less reliable than for other pixels.
    • Pixel 7 experienced an increase in broadband noise over a ~fifty-day interval estimated to extend from 5-29-2025 10:00 UTC (XRISM Mission Elapsed TIME = 202176000) to 2020-07-23 14:00 (XRISM Mission Elapsed TIME =UTC 206978000). This resulted in a degraded pixel 7 resolution (from ~4.5 eV to >7 eV and sometimes as high as 9 eV FWHM) for OBSIDs 201005010, 201099010, 201100010, 201090010, 201124010, 201113010, 201002010, 201092010, 201067010, 201094010, 201031010, 201081010, 201112010, 201086010, 201085010, 201058010 ,201042010, 201097010, 201078010.For these observations, events in this pixel ought to be completely omitted from analysis or, for the first and last of these observations, for part of the observation following the pixel partial-exposure procedures referred to in version 3.1 of the Quick-Start Guide. Note that these data remain proprietary until one-year after their delivery to the PI.
  • RSL-3: ARF
    • Resolve region file outputs from xamkregion should not be used as input regions to the ARF generator (xaarfgen or xaxmaarfgen) because the pixels in these region files are slightly rotated by different amounts relative to each other and some of the pixels overlap with each other. This is a design flaw inherited from Hitomi, based on mis-use of the XRISM ground calibration data and will be fixed in the future. However, note that the Resolve DET coordinates region file currently output from the pipeline is created with xamkregion, so should not be used as input to the ARF generator, so should only be used for overlaying Resolve FoV on images from other instruments.
    • Sub-array XMA effective area. The XMA effective area energy dependence shows some variation on the size scale of Resolve pixels, but for point sources that have < 1 ct/s (0-30 keV, all grades), the Resolve pixels beyond the central four make a diminishing contribution to the net effective area for an Hp spectrum from the whole array. However, as the source count rate increases, pixels exposed to the outer parts of the PSF make an increasing contribution to the net Hp effective area relative to pixels exposed to the PSF core. A more accurate net effective area will be obtained by making ARFs and RMFs separately for groups of pixels, and then combining the ARFs and RMFs into a single response (RSP) file, using appropriate weights for the ARFs. For example, the array could be divided into three pixel groups, corresponding to the four central pixels, an inner ring of pixels that surround the central four, and an outer ring consisting of the remaining pixels. Note that smaller regions on the detector require correspondingly larger values of the 'numphoton' parameter in xaarfgen.
  • RSL-4: RMF
    • Resolve RMF size. The Resolve X-large RMF should not be made with the full 0.5 eV per bin resolution as doing so will result in an unreasonably large size of 7 Gb. Instead, users are strongly urged to use the option to split the response function into two matrices (e.g., with command like 'rslmkrmf splitrmf=yes elcbinfac=16 whichrmf=X splitcomb=yes'). Read the help files for rslrmf and rslmkrmf for details.
    • Resolve RMF normalization and net effective area. It has been found that there is an excess of Ls (ITYPE=4) events in Resolve data that are not due to the source. In addition, for very bright sources, the observed branching ratios into different grades do not agree with theoretical expectations, possibly due to screening out of true source events. These phenomena are under investigation. They affect the calculated effective area function and the relative weights of the line-spread functions in different pixels. A related issue for bright sources is that the XMA effective area per pixel varies by such a large amount over the Resolve array, that the grade fractions vary significantly pixel-to-pixel. Large-amplitude variability in bright sources further complicates the analysis. A more complicated method of constructing response files is required and is currently in development. Also, for phase-resolved spectroscopy, different response functions may be required for different phases. Additionally, for now, it is recommended not to fit any spectra above 12 keV because branching ratio anomalies are worse at higher energies.

      For now, it is recommended to make Resolve RMFs using a custom cleaned event file that has Ls events removed, and has ALL events that have PI values lower than 6000 and higher than 20000 also removed. This custom event file is to be used for RMF generation only, and not for analysis. For sources that have more than 1 ct/s in the 0-30 keV band (summed over all 5 grades), the procedure will result in an upper limit on the effective area (corresponding to lower limits on derived fluxes). An approximate lower limit on the effective area can be obtained by making RMFs using a different custom event file that does NOT have Ls events removed, but has all events with PI<10000 and PI>16000 removed. For sources that have <1 ct/s, the lower and upper limits on the effective area functions converge. Until solutions are found for solving the anomalous Ls events and branching ratios problems, the uncertainties in effective areas have to be treated as systematic errors.
    • Pixel-to-Pixel Spectral Differences. It has been found that in some sources (including point sources), there are broad and significant differences in spectra extracted from different Resolve pixels. The magnitude and energy dependence of the differences varies from source to source and from pixel to pixel, and is not dependent on count rate. The origin of these spectral differences is not yet understood, and is under investigation. The figure below shows the ratio of spectra from 31 pixels that exclude the central four, to the spectra from the central four, for two non-variable point sources. The left panel below shows a case for which the spectral ratio is flat (so any spectral differences are smaller than the statistical errors), and the right-hand panel shows a case for which the spectral ratio changes by as much as a factor of ~2.3 between 2 and 12 keV. Until the origin of the spectral differences is understood, they will have to be treated as systematic errors. If the photon statistics are sufficiently high, spectral differences may be observed between smaller groups of pixels, or even between individual pixels. At this time, it is not known how to determine which pixels or group of pixels yield spectra that are most faithful to the "true" spectrum. The pixel-to-pixel variation of the XMA effective area is insufficient to explain the observed spectral anomalies. Moreover, an XMA origin would impose similar spectral differences for all on-axis point-source observations, but is found not to be the case. Extended sources present a greater challenge, because while it is possible to directly observe sub-array spectral differences in point-sources, for extended sources we cannot assume that the spectra in all pixels should intrinsically be essentially the same. This means that at this time it is not possible in general to gauge the severity of sub-array spectral distortion in extended sources, and this also complicates spatial-spectral mixing (SSM) analyses. In addition, it is possible that "jumps" in spectral parameters may occur between pixels that imply unphysical spatial variations.
  • RSL-5: rslpipeline
    • For software earlier than HEASoft 6.35: In rare cases, on some platforms, rslpipeline (and xapipeline) halted or stalled when running the rslctfluct task due to a memory issue. This was fixed in HEASoft 6.35, so it is recommended to update your installation if you have not done so already.
  • RSL-6: XSELECT bug for making Resolve images and lightcurves
    • For software earlier than HEASoft 6.35: There was a bug in XSELECT that ignored PI selection for making lightcurves or images (with the pha_cutoff command) if the user had also specified Resolve pixel selection using the "filter column PIXEL" command using upper case letters for the column name. The result was that the lightcurve or image was made from all events from the selected pixels, regardless of their PI values. Although a workaround is to ensure that the column name "pixel" is specified with lower case letters: in the command "filter column pixel", it is recommended to update your installation if you have not done so already.
  • RSL-7: Resolve sub-array RMFs with region file
    • For Resolve sub-array spectroscopy, if you run rslmkrmf with an input region file (as opposed to a pixel list), the RMF (and effective area for bright sources) will not be correct, unless you explicitly set the rslmkrmf input parameter 'pixeltest' to 'pixeltest=CENTER'. This is because the default for the parameter 'pixeltest' in rslmkrmf in HEASoft 6.35.2 or earlier does not give the correct region to pixel ID conversions. Check your previous analyses for the commands that you used to make Resolve RMFs for sub-array spectroscopy.
  • RSL-8: xmatraceback bug
    • There is a bug in xmatraceback (HEASoft 6.36) that affects the output "*_photon_fractions.fits" file. The bug affects numerical flux conversion factors that apply to spectral-fitting with ARFs made with image mode. In the "*_photon_fractions.fits" file, the value of the keyword FRINPIMG and the values in the column PixFracInpIMG are incorrect. The bug will be fixed in the next release. A workaround is to multiply the incorrect values by the ratio of the number of rows in the input heasim event file to the number of rows in the input raytracing file that have an energy lying between emin and emax (inclusive), where emin and emax are input parameters to xmatraceback. For example, if the raytracing file input to xmatraceback is rtfile1.fits, and the emin and emax inputs to xmatraceback were 2.0 and 10.0 respectively, make the file rtfile2.fits:

      ftcopy "raytrace_xa000126000rsl_p0px1000_hv2ax10a_img1.fits[1][(ENERGY>=2.0)&&(ENERGY <=10.0)]" rtfile2.fits

      The correction factor is the ratio of the keyword NAXIS2 in ftfile2.fits[1] to the keyword NAXIS2 in the heasim events file (extension 1) that was input to xmatraceback.

Xtend

  • XTD-1: Data format and naming conventions
    • Xtend can have multiple input files split by the operating modes. The Xtend file naming convention is:
      • xaNNNNNNNNNxtd_pRzdmcccca_cl.evt
      • xa000127000xtd_p031100010_cl.evt (example 1)
      • xa000127000xtd_p032000010_cl.evt (example 2)

      • where:
      • NNNNNNNNN = 9-digit OBSID
      • p = 'p' for pointed observation, 's' for slew observation, 'a' for all (will usually be 'p')
      • R = index for data divided into multiple files due to (e.g.) filesize. R is in the range [0-9a-zA- Z] and set to 0 if no division is needed.
      • zdmccca = dataClass, which describes the CCDs used and their configuration, where:
        • z = placeholder always set to 3 for Xtend in-flight data
        • d = DETNAM: 0 = CCD, 1 = CCD12, 2 = CCD34
        • m = DATAMODE: 0 = WINDOW1 (full window), 1 = WINDOW2 (1/8 window), 2 = WINDOW1BURST (full window + 0.1 sec burst mode), 3 = WINDOW2BURST (1/8 window + 0.1 sec burst mode)
        • cccc = hexadecimal encoding of on-board instrument settings (thresholds, area discrimination, ADC chains used, readout nodes used, charge injection settings, etc.)
        • a = reserved bit, 0 for now
    • Some common dataClass values you may encounter:
      • 30000010: the event list contains all four CCDs in full window mode
      • 31100010: the event list contains CCD1 and CCD2 in 1/8 window mode
      • 31300010: the event list contains CCD1 and CCD2 in 1/8 window + 0.1 sec burst mode
      • 32000010: the event list contains CCD3 and CCD4 in full window
      • The first three digits of this (300*, 311*, 313*, 320*) are the important ones that tell you the mode and the contents of the event list, whether it contains data from all CCDs or just a pair. You can also look at the DETNAM, DATAMODE, and DATACLAS header keywords.
      • DATAMODE = WINDOW1BURST (full window + 0.1 sec burst mode) is not supported for science observations, only for Xtend IT usage.
    • Pairs of CCDs share clock drivers, so CCD1+CCD2 always operate in the same mode, and CCD3+CCD4 always operate in the same mode.
    • CCD2 is the aimpoint CCD, and only the CCD1+CCD2 pair is allowed to operate in 1/8 window or 1/8 window+0.1 sec burst mode; the CCD3+CCD4 pair always operates in full window mode.
      • You should always see a 320* event list along with a 311* or 313* event list.
      • You will generally only see a single 300* event list when all CCDs are in full window mode.
  • XTD-2: Regions in 1/8 window mode
    • In 1/8 window mode, the region for input to the ARF generator must be confined to be completely inside the boundaries of the window, otherwise the ARF will not be correct. The image below in DET coordinates shows an example of a correct region.


  • XTD-3: Flickering pixels, cosmic-ray echo events, anomalous pixels
    • It has been found that the tool searchflickpix, used for removing events from flickering pixels and cosmic-ray echo events does not work well with XRISM data in an automated setting, so it is not applied in the XRISM pipeline. This is because there is a risk that true source events might be removed. However, the pipeline still runs searchflickpix without applying it to the data, but produces the .fpix file that lists the flagged pixels. This file is used by xaexpmap (which then is used by xaarfgen). Therefore, the .fpix file from the pipeline should not be used as input to the xaexpmap. The user should follow the Quick-Start Guide on mitigation strategies for anomalous pixel removal, and either make their own .fpix file, or specify NONE as input to xaexpmap. Note the following corrections to the Quick-Start Quide v2.1:
      • All of the searchflickpix commands should have the following parameter settings added:
        • xcol=DETX ycol=DETY
      • If you are directed to section 5 from step 3 in the Appendix (i.e., if you are applying a simple threshold cut), add the following parameter settings to both searchflickpix commands:
        • grade=ALL
  • XTD-4: Obscured portion of the field of view (updated 2025-01-23)
    • The corner of the Xtend FoV farthest from the aimpoint is blocked by a structure of the camera. While there may be counts detected in this region, they are all from unfocused non-X-ray background and not from celestial X-rays. The approximate location of this region can be seen below in a DET-coordinate image of the sunlit limb of the Earth from the 'day Earth' trend archive.


  • XTD-5: Bad pixel image missing in data processed with pipeline installed May 22, 2025
    • Recent pipeline processing has an issue whereby the bad pixel image (‘outbadimg’ in xtdflagpix) is not produced. This file is a required input to xaexpmap, so without it users cannot create exposure maps or ARFs for Xtend data.
    • Follow these instructions to see if your data is affected:
      • The issue affects data processed with pipeline version 03.01.014.010 installed May 22, 2025. The pipeline version can be found in the PROCVER header keyword in the FITS event list.
      • For each cleaned event list in <OBSID>/xtend/event_cl, for example:
      • 000127000/xtend/event_cl/xa000127000xtd_p031100010_cl.evt.gz
      • 000127000/xtend/event_cl/xa000127000xtd_p032000010_cl.evt.gz
      • There should be a corresponding bad pixel image in <OBSID>/xtend/event_uf, for example:
      • 000127000/xtend/event_uf/xa000127000xtd_p031100010.bimg.gz
      • 000127000/xtend/event_uf/xa000127000xtd_p032000010.bimg.gz
      • If the .bimg file(s) are missing, your data is affected and you will need to produce them by hand.
      • If you have reprocessed your Xtend data with xapipeline or xtdpipeline, the .bimg files should be present and you can skip the rest of the instructions.
    • Follow these instructions to create the .bimg file(s):
      • cd 000127000/xtend/event_uf
      • punlearn xtdflagpix
      • xtdflagpix \
        • infile=xa000127000xtd_p031100010_uf.evt \
        • outfile=xtdflagpix_tmp.evt \
        • outbadimg=xa000127000xtd_p031100010.bimg \
        • hotpixfile=xa000127000xtd_a031100010.hpix \
        • flickpixfile=NONE
      • rm xtdflagpix_tmp.evt
      • Replace instances of 000127000 with the OBSID for your observation.
      • Replace the 031100010 with the relevant 8-character string from your event list name.
      • Run this for each cleaned event list name, but use the unfiltered (‘uf’) event list as the input to xtdflagpix.
      • Delete the output event list after running xtdflagpix, since you do not need it.
      • When you run xaexpmap, set the ‘badimgfile’ parameter to the .bimg file you created here.

Miscellaneous

  • MSC-1: The SDC has prepared ds9 'template' region files that include the XRISM instrument fields of view. These can be displayed on any image with WCS information, e.g. from ROSAT, Chandra, XMM, eROSITA, DSS, SDSS. These regions should be used only for display purposes, not for extracting products for analysis.