next up previous contents
Next: 4. Setting Up and Up: XMM ABC Guide Previous: 2. Useful Information and   Contents

Subsections


3. Data


3.1 USEFUL DOCUMENTATION

There are a number of documents which the users of XMM-Newton data should be aware of. These documents include the SSC Products Specification, Data Files Handbook, Reading Data Products CD's (the most recent versions of these documents can be found in the SOC Document section under
http://xmm.esac.esa.int/external/xmm_user_support/documentation/uhb_frame.shtml), and the SAS Users Guide ( http://xmm.esac.esa.int/external/xmm_user_support/documentation/sas_usg/USG/).

For observation data sets going to US PIs, the GOF makes the data available online after PGP encryption and after converting the file names to upper case. When the proprietary period for the observation expires the data are decrypted leaving the file names unchanged. A simple decryption script, minus the relevant keys of course, can be found at:
ftp://legacy.gsfc.nasa.gov/xmm/software/decrypt.pl.
NOTE: Laura Brenneman wrote a script and accompanying help file that gives explicit directions on how to most quickly pull over all the files in a data set from the archive, as well as decrypting, and uncompressing the files in preparation for data analysis. This package can be found at:
ftp://legacy.gsfc.nasa.gov/xmm/software/prepare_xmm_data.tar.gz. and contains the following files: README, decrpyt.pl, and prepare_xmm_data.pl.


3.2 THE DATA

One of the first steps that should be taken when examining your data is to check to see what you actually have. XMM-Newton observations can be broken into several exposures which are each assigned separate observation numbers. These separate exposures can be radically different in length and can also have the different instruments in different modes. For example, in one case the full observation was 60 ks with EPIC and RGS active but there was one delivered exposure which was $\sim3$ ks and had only RGS active. (This can happen because the RGS can operate farther into regions of higher radiation than the EPIC detector. The additional observation time can be considered an additional exposure with only the RGS active.) Two files are useful for this examination. First, the primary HTML page is INDEX.HTM which is included in the Pipeline Products. This page lists basic information for the observation plus the operational modes, filters, and exposure start and stop times for the individual instruments. It also has links to various summary pages, including those for the instruments. (In the case above, the EPIC summary page simply stated that ``EPIC exposures processed by PPS None.'') Specifically, LOOK at the P*SUMMAR0000.HTM files in the pipeline products (easily available through the links). Second, to quickly access images from the various instruments, examine the PPSGRA Pipeline Products page, viewable with a web browser. These files have the nomenclature P*PPSGRA000_0.HTM. (see § 3.3.3).


3.3 PI Data

Proprietary XMM-Newton data is available for download via your XSA account. Email instructions from the XMM-SOC at Vilspa are sent to the address on record with detailed directions on how to retrieve your data via the XSA.

The data files can be considered to come in two groups in separate subdirectories when retrieved, the Observation Data Files (ODF) files and Pipeline Processing (PPS) files. The ODF data contain all of the observation-specific data necessary for reprocessing the observation. The PPS data contain, among other things, calibrated photon event files and source lists.

For observation data sets going to US PIs, the GSFC GOF makes the data available in two directories containing the following groups of files.


3.3.1 ODF Data

ODF data come with file names in the following format:

NOTE: For SAS processing, the file names should contain all upper case characters. However, at least with early CDs, the file names used lower case characters. The GSFC XMM-Newton GOF provides a script to rename the files.


Table 3.1: ODF data file identifiers.
Data ID Contents
  EPIC files
IME Event list for individual CCDs, imaging mode
RIE Event list for individual CCDs, reduced imaging mode
CTE Event list for individual CCDs, compressed timing mode
TIE Event list for individual CCDs, timing mode
BUE Event list for individual CCDs, burst mode
AUX Auxiliary file
CCX Counting cycle report (auxiliary file)
HBH HBR buffer size, non-periodic housekeeping
HCH HBR configuration, non-periodic housekeeping
HTH HBR threshold values, non-periodic housekeeping
PEH Periodic housekeeping
PTH Bright pixel table, non-periodic housekeeping
DLI Discarded lines data
PAH Additional periodic housekeeping
PMH Main periodic housekeeping
  RGS files
AUX on-board processing statistics
SPE raw event list for one CCD
DII diagnostic images
D1H CCD readout settings
D2H CCD readout settings
PFH housekeeping data
ODX pixel offset data
  XMM files
ATS spacecraft attitude history
  OM files
IMI imaging file
THX tracking history file
WDX window data auxiliary file
NPH non-periodic housekeeping file
PEH periodic housekeeping file
PAX field acquisition data
RFX priority reference frame data
PFX priority fast mode data
FAE event list (if fast mode was used)
   



3.3.2 Pipeline Product Data - Summary Files and Groupings

The pipeline processing produces quite a number of useful products which allow a first look at the data, but can overwhelm the user by their sheer numbers. The first place to look is the INDEX.HTM page which organizes the presentation of the data and provides links to other PP pages. The INDEX.HTM page also lists general observation information (target, date, time, etc.) and instrument modes.

The INDEX.HTM page provides links to various observation summary pages, which have names with the following nomenclature:

PP data contain some immediately useful data products such as calibrated photon event lists, source lists, and images. While there are a large number of products which come in a single directory, they can be associated in up to 15 groupings; see Table 3.2. (The number of groups can vary depending on the number of operational instruments, e.g., if the OM is turned off there are no OM products.) Further information on each of these groupings and associated files, such as file contents, file types, and how they may be viewed, can be found in Table 3.3. Each group has an associated HTML file which organizes access to the files and provides a limited description of them. The names of the HTML files are of the following form:


Table 3.2: Pipeline Processing groupings.
Group ID Contents
  PP files
PPSDAT Contains the Calibration Index File (CIF) used in the pipeline processing (*CALIND*),
  PPS information, and the attitude history time series (*ATTTSR*) in gzipped FITS
  or ASCII format.
PPSGRA Contains the OM tracking history plots, PPS, EPIC, OM, RGS observation, and PPS
  run summaries. NOTE: CHECK THESE OUT
PPSMSG ASCII file containing pipeline processing report
  EPIC files
CRSCOR Contains PDF files of POSS II finding charts, HTML files of cross correlations
  with the SIMBAD data base, FITS tables for the detected sources
EANCIL Contains the exposure maps in a variety of energy bands and the source-detection
  sensitivity maps for the EPIC instruments. The sensitivities are in units of
  counts s$^{-1}$ corrected for vignetting and corresponding to a likelihood specified
  in the FITS header. The files are gzipped with a .FTZ extension.
EEVLIS Contains calibrated photon event files for the EPIC detectors. If the files are
  sufficiently large they may be separated into two tar files. The files are gzipped
  fits files with a .FTZ extension.
ESKYIM This group contains the event images in a variety of energy bands. The fits files
  are gzipped with a .FTZ extension, the full images also come as PNG images.
ESRLIS Contains EPIC observation source lists. There is an HTML page of the merged
  source list and gzipped fits tables of source lists from the different instruments
  and source detection tasks.
  OM files
OIMAGE Contains OM sky images in gzipped FITS format.
OMSLIS Contains OM observation source lists in gzipped FITS format.
OMSRTS Contains OM star tracking time series in gzipped FITS format.
  RGS files
REVLIS Contains the RGS source and event lists in gzipped FITS format
REXPIM Contains the RGS exposure maps in gzipped FITS format
RIMAGE Contains the RGS images (both energy dispersion and cross dispersion) in gzipped
  FITS and PNG formats
RSPECT Contains the RGS source and background spectra in gzipped FITS and PDF formats



3.3.3 Pipeline Product Data - Data Files

The data file names are of the form (see Table 41 in the XMM Data Files Handbook):


Table 3.3: Pipeline Processing data files
Group ID File ID Contents File Type View With
    EPIC files    
CRSCOR FCHART Finding chart PDF Acrobat Reader
  ROSIMG ROSAT image of region PDF Acrobat Reader
  SNNNNN$^1$ Source cross-correlation results Zipped FITS fv
  DNNNNN$^1$ Catalog descriptions PDF Acrobat Reader
  FNNNNN$^1$ FOV cross-correlation result Zipped FITS fv
         
ESKYIM IMAGE_8 Sky image 0.2 - 12.0 keV Zipped FITS ds9, Ximage, fv
  IMAGE_1 Sky image 0.2 - 0.5 keV Zipped FITS ds9, Ximage, fv
  IMAGE_2 Sky image 0.5 - 2.0 keV Zipped FITS ds9, Ximage, fv
  IMAGE_3 Sky image 2.0 - 4.5 keV Zipped FITS ds9, Ximage, fv
  IMAGE_4 Sky image 4.5 - 7.5 keV Zipped FITS ds9, Ximage, fv
  IMAGE_5 Sky image 7.5 - 12.0 keV Zipped FITS ds9, Ximage, fv
         
EANCIL EXPMAP_8 Exposure map 0.2 - 12.0 keV Zipped FITS, PNG ds9, Ximage, fv, web browser
  EXPMAP_1 Exposure map 0.2 - 0.5 keV Zipped FITS ds9, Ximage, fv
  EXPMAP_2 Exposure map 0.5 - 2.0 keV Zipped FITS ds9, Ximage, fv
  EXPMAP_3 Exposure map 2.0 - 4.5 keV Zipped FITS ds9, Ximage, fv
  EXPMAP_4 Exposure map 4.5 - 7.5 keV Zipped FITS ds9, Ximage, fv
  EXPMAP_5 Exposure map 7.5 - 12.0 keV Zipped FITS ds9, Ximage, fv
         
EEVLIS$^2$ MIEVLI MOS imaging mode event list Zipped FITS xmmselect, fv, Xselect
  PIEVLI PN imaging mode event list Zipped FITS xmmselect, fv, Xselect
  TIEVLI PN, MOS timing mode event list Zipped FITS xmmselect, fv, Xselect
         
ESRLIS EBLSLI Box-local detect source list Zipped FITS fv
  EBMSLI Box-map detect source list Zipped FITS fv
  EMSRLI Max-like detect source list Zipped FITS fv
  OBSMLI Summary source list Zipped FITS, HTML fv, web browser
    RGS files    
REVLIS SRCLI_ RGS Source Lists Zipped FITS fv
  EVENLI RGS Event lists Zipped FITS xmmselect, fv
         
REXPIM EXPMAP RGS Exposure Maps Zipped FITS ds9, Ximage, fv
         
RSPECT SRSPEC1 1st Order Source Spectra Zipped FITS Xspec, fv
  SRSPEC2 2nd Order Source Spectra Zipped FITS Xspec, fv
  BGSPEC1 1st Order Background Spectra Zipped FITS Xspec, fv
  BGSPEC2 2nd Order Background Spectra Zipped FITS Xspec, fv
  SRSPEC Spectra Plots PDF format Acrobat Reader
         
RIMAGE ORDIMG Images, disp. vs. X-disp Zipped FITS, PNG ds9, Ximage, fv, web browser
  IMAGE_ Images, disp. vs. PI Zipped FITS, PNG ds9, Ximage, fv, web browser
    OM files    
OIMAGE SIMAGE OM Sky Image Gzipped FITS ds9, Ximage, fv
         
OMSLIS SWSRLI OM Source Lists Zipped FITS fv
         
OMSRTS TSTRTS Tracing Star Time Series Zipped FITS fv

$^1$ NNNNN - Alphanumeric ID
$^2$ Files for only those modes which were active will be included


next up previous contents
Next: 4. Setting Up and Up: XMM ABC Guide Previous: 2. Useful Information and   Contents
Lynne Valencic 2014-04-17