3. Data

3.1 Useful Documentation

There are a number of documents which the users of XMM-Newton data should be aware of. These documents include the Data Files Handbook and Pipeline Products Description. The most recent versions of these documents can be found in the SOC Document section under
https://www.cosmos.esa.int/web/xmm-newton/documentation.

For observation data sets going to US PIs, the GOF makes the data available online after PGP encryption and after converting the file names to upper case. When the proprietary period for the observation expires the data are decrypted leaving the file names unchanged. A simple decryption script, minus the relevant keys of course, can be found at:
https://heasarc.gsfc.nasa.gov/FTP/xmm/software/decrypt.pl.
NOTE: Laura Brenneman wrote a script and accompanying help file that gives explicit directions on how to most quickly pull over all the files in a data set from the archive, as well as decrypting, and uncompressing the files in preparation for data analysis. This package can be found at:
https://heasarc.gsfc.nasa.gov/FTP/xmm/software/prepare_xmm_data.tar.gz. and contains the following files: README, decrpyt.pl, and prepare_xmm_data.pl.

3.2 The Data

One of the first steps that should be taken when examining your data is to check to see what you actually have. XMM-Newton observations can be broken into several exposures which are each assigned separate observation numbers. These separate exposures can be radically different in length and can also have the different instruments in different modes. For example, in one case the full observation was 60 ks with EPIC and RGS active but there was one delivered exposure which was $\sim3$ ks and had only RGS active. (This can happen because the RGS can operate farther into regions of higher radiation than the EPIC detector. The additional observation time can be considered an additional exposure with only the RGS active.) The instrument summary pages, with nomenclature P*SUMMAR0000.HTM, are a good place to start. Second, to quickly access images from the various instruments, examine the PPSGRA Pipeline Products page, viewable with a web browser. These files have the nomenclature P*PPSGRA000_0.HTM. (see § 3.3.3).

3.3 PI Data

Proprietary XMM-Newton data is available for download via your XSA account. Email instructions from the XMM-SOC in Europe are sent to the address on record with detailed directions on how to retrieve your data via the XSA.

The data files can be considered to come in two groups in separate subdirectories when retrieved, the Observation Data Files (ODF) files and Pipeline Processing (PPS) files. The ODF data contain all of the observation-specific data necessary for reprocessing the observation. The PPS data contain, among other things, calibrated photon event files and source lists.

For observation data sets going to US PIs, the GSFC GOF makes the data available in two directories containing the following groups of files.

• ODF - The ODF (raw) data files
• PPS - The pipeline processed data products

3.3.1 ODF Data

ODF data come with file names in the following format:

• mmmm_iiiiiijjkk_aabeeeccfff.zzz

  mmmm - revolution orbit number

  iiiiii - proposal number

  jj - target ID number in proposal

  kk - exposure number for target

  (NOTE: The ten-digit combination of iiiiiijjkk is the observation number and is used repetitively throughout the file nomenclature.)

  aa - detector (M1 - MOS1, M2 - MOS2, PN - PN, OM - OM, R1 - RGS1, R2 - RGS2, SC - spacecraft)

  b - flag for scheduled (S) or unscheduled (U) observations, or general purpose X files

  eee - exposure number

  cc - CCD number or OM window number

  fff - data identifier for the three detectors or spacecraft itself; see Table 3.1.

  zzz - format (FIT - FITS, ASC - ASCII)

NOTE: For SAS processing, the file names should contain all upper case characters. However, at least with early CDs, the file names used lower case characters. The GSFC XMM-Newton GOF provides a script to rename the files.

Table 3.1: ODF data file identifiers.

Data ID	Contents
	EPIC files
IME	Event list for individual CCDs, imaging mode
RIE	Event list for individual CCDs, reduced imaging mode
CTE	Event list for individual CCDs, compressed timing mode
TIE	Event list for individual CCDs, timing mode
BUE	Event list for individual CCDs, burst mode
AUX	Auxiliary file
CCX	Counting cycle report (auxiliary file)
HBH	HBR buffer size, non-periodic housekeeping
HCH	HBR configuration, non-periodic housekeeping
HTH	HBR threshold values, non-periodic housekeeping
PEH	Periodic housekeeping
PTH	Bright pixel table, non-periodic housekeeping
DLI	Discarded lines data
PAH	Additional periodic housekeeping
PMH	Main periodic housekeeping
	RGS files
AUX	on-board processing statistics
SPE	raw event list for one CCD
DII	diagnostic images
D1H	CCD readout settings
D2H	CCD readout settings
PFH	housekeeping data
ODX	pixel offset data
	XMM files
ATS	spacecraft attitude history
	OM files
IMI	imaging file
THX	tracking history file
WDX	window data auxiliary file
NPH	non-periodic housekeeping file
PEH	periodic housekeeping file
PAX	field acquisition data
RFX	priority reference frame data
PFX	priority fast mode data
FAE	event list (if fast mode was used)

3.3.2 Pipeline Product Data - Summary Files and Groupings

The pipeline processing produces quite a number of useful products which allow a first look at the data, but can overwhelm the user by their sheer numbers. The first place to look is the summary files, which have names with the following nomenclature:

• PPiiiiiijjkkAAX000SUMMAR0000.HTM

  iiiiiijjkk - observation number

  AA - detector ID (EP - EPIC, OM - Optical Monitor, RG - RGS, OB - Observation)

PP data contain some immediately useful data products such as calibrated photon event lists, source lists, and images. While there are a large number of products which come in a single directory, they can be associated in up to 15 groupings; see Table 3.2. (The number of groups can vary depending on the number of operational instruments, e.g., if the OM is turned off there are no OM products.) Further information on each of these groupings and associated files, such as file contents, file types, and how they may be viewed, can be found in Table 3.3. Each group has an associated HTML file which organizes access to the files and provides a limited description of them. The names of the HTML files are of the following form:

• PPiiiiiijjkkAAAAAA000_0.HTM

  iiiiiijjkk - observation number

  AAAAAA - group identifier (see Table 3.2)

Table 3.2: Pipeline Processing groupings.

Group ID	Contents
	PP files
PPSDAT	Contains the Calibration Index File (CIF) used in the pipeline processing (*CALIND*),
	PPS information, and the attitude history time series (*ATTTSR*) in gzipped FITS
	or ASCII format.
PPSGRA	Contains the time series plots, images, spectra, for EPIC, OM, RGS observations, and PPS
	run and observation summaries.
PPSMSG	ASCII file containing pipeline processing report
	EPIC files
CRSCOR	Contains PDF files of POSS II finding charts, HTML files of cross correlations
	with the SIMBAD data base, FITS tables for the detected sources
EANCIL	Contains the exposure maps in a variety of energy bands and the source-detection
	sensitivity maps for the EPIC instruments. The sensitivities are in units of
	counts s$^{-1}$ corrected for vignetting and corresponding to a likelihood specified
	in the FITS header. The files are gzipped with a .FTZ extension.
EEVLIS	Contains calibrated photon event files for the EPIC detectors. If the files are
	sufficiently large they may be separated into two tar files. The files are gzipped
	fits files with a .FTZ extension.
ESKYIM	This group contains the event images in a variety of energy bands. The fits files
	are gzipped with a .FTZ extension.
ESRLIS	Contains EPIC observation source lists. There is an HTML page of the merged
	source list and gzipped fits tables of source lists from the different instruments
	and source detection tasks.
	OM files
OIMAGE	Contains OM sky images in gzipped FITS format (not to be confused
	with the *OIMAGE*, PNG, or FTZ files, which are EPIC images.)
OMSLIS	Contains OM observation source lists in gzipped FITS format.
	RGS files
REVLIS	Contains the RGS source and event lists in gzipped FITS format
REXPIM	Contains the RGS exposure maps in gzipped FITS format
RIMAGE	Contains the RGS images in gzipped FITS format
RSPECT	Contains the RGS source and background spectra and affiliated files
	in gzipped FITS format

3.3.3 Pipeline Product Data - Data Files

The data file names are of the form (see Table 41 in the XMM Data Files Handbook):

• PiiiiiijjkkaablllCCCCCCnmmm.zzz

  iiiiiijjkk - observation number

  aa - detector, M1 - MOS1, M2 - MOS2, PN - PN, CA - for files from the CRSCOR group, R1 - RGS1, R2 - RGS2, OM - OM.

  b - flag for scheduled (S) or unscheduled (U) observations, or X for files from the CRSCOR group (and any product that is not due to a single exposure)

  lll - exposure number

  CCCCCC - file identification for data from each detector; see Table 3.3

  n - For EPIC data, this is the exposure map band number; for RGS data, this is the spectral order number; for the OM, this is the OM window within the exposure.

  mmm - source number in hexidecimal

  zzz - file type (e.g., PDF, PNG, FTZ, HTM)

  ASC - ASCII file, use a web browser, or the ``more'' command

  ASZ - gzipped ASCII file

  FTZ - gzipped FITS format, use ds9, Ximage, Xselect, fv

  HTM - HTML file, use Firefox or other web browser

  PDF - Portable Data Format, use Acrobat Reader

  PNG - Portable Networks Graphics file, use a web browser

  TAR - TAR file

Table 3.3: Pipeline Processing data files

Group ID	File ID	Contents	File Type	View With
		EPIC files
CRSCOR	FCHART	Finding chart	PDF	Acrobat Reader
	ROSIMG	ROSAT image of region	PDF	Acrobat Reader
	SNNNNN$^1$	Source cross-correlation results	Zipped FITS	fv
	DNNNNN$^1$	Catalog descriptions	HTML	web browser
	FNNNNN$^1$	FOV cross-correlation result	Zipped FITS	fv
ESKYIM	IMAGE_8	Sky image 0.2 - 12.0 keV	Zipped FITS	ds9, Ximage, fv
	IMAGE_1	Sky image 0.2 - 0.5 keV	Zipped FITS	ds9, Ximage, fv
	IMAGE_2	Sky image 0.5 - 2.0 keV	Zipped FITS	ds9, Ximage, fv
	IMAGE_3	Sky image 2.0 - 4.5 keV	Zipped FITS	ds9, Ximage, fv
	IMAGE_4	Sky image 4.5 - 7.5 keV	Zipped FITS	ds9, Ximage, fv
	IMAGE_5	Sky image 7.5 - 12.0 keV	Zipped FITS	ds9, Ximage, fv
EANCIL	EXPMAP8	Exposure map 0.2 - 12.0 keV	Zipped FITS	ds9, Ximage, fv
	EXPMAP1	Exposure map 0.2 - 0.5 keV	Zipped FITS	ds9, Ximage, fv
	EXPMAP2	Exposure map 0.5 - 2.0 keV	Zipped FITS	ds9, Ximage, fv
	EXPMAP3	Exposure map 2.0 - 4.5 keV	Zipped FITS	ds9, Ximage, fv
	EXPMAP4	Exposure map 4.5 - 7.5 keV	Zipped FITS	ds9, Ximage, fv
	EXPMAP5	Exposure map 7.5 - 12.0 keV	Zipped FITS	ds9, Ximage, fv
EEVLIS$^2$	MIEVLI	MOS imaging mode event list	Zipped FITS	ds9, fv, Xselect
	PIEVLI	PN imaging mode event list	Zipped FITS	ds9, fv, Xselect
	TIEVLI	PN, MOS timing mode event list	Zipped FITS	ds9, fv, Xselect
ESRLIS	OBLSLI	Box-local detect source list	Zipped FITS	fv
	OBMSLI	Box-map detect source list	Zipped FITS	fv
	OMSRLI	Max-like detect source list	Zipped FITS	fv
	OBSMLI	Summary source list	Zipped FITS, HTML	fv, web browser
		RGS files
REVLIS	SRCLI_	RGS Source Lists	Zipped FITS	fv
	EVENLI	RGS Event lists	Zipped FITS	ds9, fv
REXPIM	EXPMAP	RGS Exposure Maps	Zipped FITS	ds9, Ximage, fv
RSPECT	SRSPEC1	1st Order Source Spectrum	Zipped FITS	ds9, Xspec, fv
	SRSPEC2	2nd Order Source Spectrum	Zipped FITS	ds9, Xspec, fv
	BGSPEC1	1st Order Background Spectrum	Zipped FITS	ds9, Xspec, fv
	BGSPEC2	2nd Order Background Spectrum	Zipped FITS	ds9, Xspec, fv
	SBSPEC1	1st Order Source+Background Spectrum	Zipped FITS	ds9, Xspec, fv
	SBSPEC2	2nd Order Source+Background Spectrum	Zipped FITS	ds9, Xspec, fv
RIMAGE	ORDIMG	Images, disp. vs. X-disp	Zipped FITS	ds9, Ximage, fv
	IMAGE_	Images, disp. vs. PI	Zipped FITS	Ximage, fv
		OM files
OIMAGE	SIMAGE	OM Sky Image	Zipped FITS	ds9, Ximage, fv
OMSLIS	SWSRLI	OM Source Lists	Zipped FITS	fv

$^1$ NNNNN - Alphanumeric ID
$^2$ Files for only those modes which were active will be included