next up previous contents
Next: 4. Setting Up and Up: XMM ABC Guide Previous: 2. Useful Information and   Contents

Subsections


3. Data


3.1 Useful Documentation

There are a number of documents which the users of XMM-Newton data should be aware of. These documents include the Data Files Handbook and Pipeline Products Description. The most recent versions of these documents can be found in the SOC Document section under
https://www.cosmos.esa.int/web/xmm-newton/documentation.

For observation data sets going to US PIs, the GOF makes the data available online after PGP encryption and after converting the file names to upper case. When the proprietary period for the observation expires the data are decrypted leaving the file names unchanged. A simple decryption script, minus the relevant keys of course, can be found at:
https://heasarc.gsfc.nasa.gov/FTP/xmm/software/decrypt.pl.
NOTE: Laura Brenneman wrote a script and accompanying help file that gives explicit directions on how to most quickly pull over all the files in a data set from the archive, as well as decrypting, and uncompressing the files in preparation for data analysis. This package can be found at:
https://heasarc.gsfc.nasa.gov/FTP/xmm/software/prepare_xmm_data.tar.gz. and contains the following files: README, decrpyt.pl, and prepare_xmm_data.pl.


3.2 The Data

One of the first steps that should be taken when examining your data is to check to see what you actually have. XMM-Newton observations can be broken into several exposures which are each assigned separate observation numbers. These separate exposures can be radically different in length and can also have the different instruments in different modes. For example, in one case the full observation was 60 ks with EPIC and RGS active but there was one delivered exposure which was $\sim3$ ks and had only RGS active. (This can happen because the RGS can operate farther into regions of higher radiation than the EPIC detector. The additional observation time can be considered an additional exposure with only the RGS active.) The instrument summary pages, with nomenclature P*SUMMAR0000.HTM, are a good place to start. Second, to quickly access images from the various instruments, examine the PPSGRA Pipeline Products page, viewable with a web browser. These files have the nomenclature P*PPSGRA000_0.HTM. (see § 3.3.3).


3.3 PI Data

Proprietary XMM-Newton data is available for download via your XSA account. Email instructions from the XMM-SOC in Europe are sent to the address on record with detailed directions on how to retrieve your data via the XSA.

The data files can be considered to come in two groups in separate subdirectories when retrieved, the Observation Data Files (ODF) files and Pipeline Processing (PPS) files. The ODF data contain all of the observation-specific data necessary for reprocessing the observation. The PPS data contain, among other things, calibrated photon event files and source lists.

For observation data sets going to US PIs, the GSFC GOF makes the data available in two directories containing the following groups of files.


3.3.1 ODF Data

ODF data come with file names in the following format:

NOTE: For SAS processing, the file names should contain all upper case characters. However, at least with early CDs, the file names used lower case characters. The GSFC XMM-Newton GOF provides a script to rename the files.


Table 3.1: ODF data file identifiers.
Data ID Contents
  EPIC files
IME Event list for individual CCDs, imaging mode
RIE Event list for individual CCDs, reduced imaging mode
CTE Event list for individual CCDs, compressed timing mode
TIE Event list for individual CCDs, timing mode
BUE Event list for individual CCDs, burst mode
AUX Auxiliary file
CCX Counting cycle report (auxiliary file)
HBH HBR buffer size, non-periodic housekeeping
HCH HBR configuration, non-periodic housekeeping
HTH HBR threshold values, non-periodic housekeeping
PEH Periodic housekeeping
PTH Bright pixel table, non-periodic housekeeping
DLI Discarded lines data
PAH Additional periodic housekeeping
PMH Main periodic housekeeping
  RGS files
AUX on-board processing statistics
SPE raw event list for one CCD
DII diagnostic images
D1H CCD readout settings
D2H CCD readout settings
PFH housekeeping data
ODX pixel offset data
  XMM files
ATS spacecraft attitude history
  OM files
IMI imaging file
THX tracking history file
WDX window data auxiliary file
NPH non-periodic housekeeping file
PEH periodic housekeeping file
PAX field acquisition data
RFX priority reference frame data
PFX priority fast mode data
FAE event list (if fast mode was used)
   



3.3.2 Pipeline Product Data - Summary Files and Groupings

The pipeline processing produces quite a number of useful products which allow a first look at the data, but can overwhelm the user by their sheer numbers. The first place to look is the summary files, which have names with the following nomenclature:

PP data contain some immediately useful data products such as calibrated photon event lists, source lists, and images. While there are a large number of products which come in a single directory, they can be associated in up to 15 groupings; see Table 3.2. (The number of groups can vary depending on the number of operational instruments, e.g., if the OM is turned off there are no OM products.) Further information on each of these groupings and associated files, such as file contents, file types, and how they may be viewed, can be found in Table 3.3. Each group has an associated HTML file which organizes access to the files and provides a limited description of them. The names of the HTML files are of the following form:


Table 3.2: Pipeline Processing groupings.
Group ID Contents
  PP files
PPSDAT Contains the Calibration Index File (CIF) used in the pipeline processing (*CALIND*),
  PPS information, and the attitude history time series (*ATTTSR*) in gzipped FITS
  or ASCII format.
PPSGRA Contains the time series plots, images, spectra, for EPIC, OM, RGS observations, and PPS
  run and observation summaries.
PPSMSG ASCII file containing pipeline processing report
  EPIC files
CRSCOR Contains PDF files of POSS II finding charts, HTML files of cross correlations
  with the SIMBAD data base, FITS tables for the detected sources
EANCIL Contains the exposure maps in a variety of energy bands and the source-detection
  sensitivity maps for the EPIC instruments. The sensitivities are in units of
  counts s$^{-1}$ corrected for vignetting and corresponding to a likelihood specified
  in the FITS header. The files are gzipped with a .FTZ extension.
EEVLIS Contains calibrated photon event files for the EPIC detectors. If the files are
  sufficiently large they may be separated into two tar files. The files are gzipped
  fits files with a .FTZ extension.
ESKYIM This group contains the event images in a variety of energy bands. The fits files
  are gzipped with a .FTZ extension.
ESRLIS Contains EPIC observation source lists. There is an HTML page of the merged
  source list and gzipped fits tables of source lists from the different instruments
  and source detection tasks.
  OM files
OIMAGE Contains OM sky images in gzipped FITS format (not to be confused
  with the *OIMAGE*, PNG, or FTZ files, which are EPIC images.)
OMSLIS Contains OM observation source lists in gzipped FITS format.
  RGS files
REVLIS Contains the RGS source and event lists in gzipped FITS format
REXPIM Contains the RGS exposure maps in gzipped FITS format
RIMAGE Contains the RGS images in gzipped FITS format
RSPECT Contains the RGS source and background spectra and affiliated files
  in gzipped FITS format



3.3.3 Pipeline Product Data - Data Files

The data file names are of the form (see Table 41 in the XMM Data Files Handbook):


Table 3.3: Pipeline Processing data files
Group ID File ID Contents File Type View With
    EPIC files    
CRSCOR FCHART Finding chart PDF Acrobat Reader
  ROSIMG ROSAT image of region PDF Acrobat Reader
  SNNNNN$^1$ Source cross-correlation results Zipped FITS fv
  DNNNNN$^1$ Catalog descriptions HTML web browser
  FNNNNN$^1$ FOV cross-correlation result Zipped FITS fv
         
ESKYIM IMAGE_8 Sky image 0.2 - 12.0 keV Zipped FITS ds9, Ximage, fv
  IMAGE_1 Sky image 0.2 - 0.5 keV Zipped FITS ds9, Ximage, fv
  IMAGE_2 Sky image 0.5 - 2.0 keV Zipped FITS ds9, Ximage, fv
  IMAGE_3 Sky image 2.0 - 4.5 keV Zipped FITS ds9, Ximage, fv
  IMAGE_4 Sky image 4.5 - 7.5 keV Zipped FITS ds9, Ximage, fv
  IMAGE_5 Sky image 7.5 - 12.0 keV Zipped FITS ds9, Ximage, fv
         
EANCIL EXPMAP8 Exposure map 0.2 - 12.0 keV Zipped FITS ds9, Ximage, fv
  EXPMAP1 Exposure map 0.2 - 0.5 keV Zipped FITS ds9, Ximage, fv
  EXPMAP2 Exposure map 0.5 - 2.0 keV Zipped FITS ds9, Ximage, fv
  EXPMAP3 Exposure map 2.0 - 4.5 keV Zipped FITS ds9, Ximage, fv
  EXPMAP4 Exposure map 4.5 - 7.5 keV Zipped FITS ds9, Ximage, fv
  EXPMAP5 Exposure map 7.5 - 12.0 keV Zipped FITS ds9, Ximage, fv
         
EEVLIS$^2$ MIEVLI MOS imaging mode event list Zipped FITS ds9, fv, Xselect
  PIEVLI PN imaging mode event list Zipped FITS ds9, fv, Xselect
  TIEVLI PN, MOS timing mode event list Zipped FITS ds9, fv, Xselect
         
ESRLIS OBLSLI Box-local detect source list Zipped FITS fv
  OBMSLI Box-map detect source list Zipped FITS fv
  OMSRLI Max-like detect source list Zipped FITS fv
  OBSMLI Summary source list Zipped FITS, HTML fv, web browser
    RGS files    
REVLIS SRCLI_ RGS Source Lists Zipped FITS fv
  EVENLI RGS Event lists Zipped FITS ds9, fv
         
REXPIM EXPMAP RGS Exposure Maps Zipped FITS ds9, Ximage, fv
         
RSPECT SRSPEC1 1st Order Source Spectrum Zipped FITS ds9, Xspec, fv
  SRSPEC2 2nd Order Source Spectrum Zipped FITS ds9, Xspec, fv
  BGSPEC1 1st Order Background Spectrum Zipped FITS ds9, Xspec, fv
  BGSPEC2 2nd Order Background Spectrum Zipped FITS ds9, Xspec, fv
  SBSPEC1 1st Order Source+Background Spectrum Zipped FITS ds9, Xspec, fv
  SBSPEC2 2nd Order Source+Background Spectrum Zipped FITS ds9, Xspec, fv
         
RIMAGE ORDIMG Images, disp. vs. X-disp Zipped FITS ds9, Ximage, fv
  IMAGE_ Images, disp. vs. PI Zipped FITS Ximage, fv
    OM files    
OIMAGE SIMAGE OM Sky Image Zipped FITS ds9, Ximage, fv
         
OMSLIS SWSRLI OM Source Lists Zipped FITS fv
         

$^1$ NNNNN - Alphanumeric ID
$^2$ Files for only those modes which were active will be included


next up previous contents
Next: 4. Setting Up and Up: XMM ABC Guide Previous: 2. Useful Information and   Contents
Lynne Valencic 2023-06-29