8. An OM Data Processing and Analysis Primer

As with EPIC and RGS datasets, many files are associated with an OM dataset. The INDEX.HTM file, and links therein, are viewable with a web browser and will help you navigate the dataset. The different types of files are discussed in Chapter 3; however, since the OM is somewhat different from the other instruments on-board XMM-Newton, we will discuss them in more detail in §8.1.

The OM can operate in IMAGING, FAST, and GRISM mode. Each of these modes has dedicated chain commands to reprocess the data: omichain, omfchain, and omgchain. These each call several procedures that are used to prepare the data for processing, make and apply flatfield images, and detect sources. The tasks omichain and omfchain also calculate the instrumental magnitudes of sources, find the position of the sources (in equatorial coordinates), and produce a sky image; omgchain produces a spectrum. If you run these chains, it is helpful to inspect the sas_log file to get a detailed list of the performed tasks. These chains rely on filters specified by the user; if no arguments are given, they run on all the files present in the ODF directory.

Most OM data are obtained in IMAGING mode. If they were obtained in the FAST mode, there will be an additional event list file corresponding to the Fast window (*FAE.FIT). §8.2 discusses in detail how the chains work. Reprocessing of data taken in FAST mode is discussed in §8.2.9. Reprocessing OM Grism data is discussed in §8.2.10.

All OM images are affected by the so-called ``stray-light'' problem (see Fig. 8.1). This problem does not affect source detection and magnitude determination but contributes to a higher background (and an ugly appearance of the images). The stray-light problem is less noticeable at UV wavelengths. A (proprietary) program to produce clean images exists but the results are strictly for display purposes only since the routine does not conserve flux. Because the stray-light problem is mainly aesthetic, there are no plans to develop publicly available routines to deal with it.

As usual, you are strongly encouraged to keep all reprocessed data in a separate directory! SAS places output files in whichever directory it is in when a task is called. We will assume that the Pipeline Processed data are in the PPS directory, the ODF data (with upper case file names, and uncompressed) are in the directory ODF, and the analysis is taking place in the PROC directory. It is also assumed that the data were prepared for processing (see §5 regardless of whether they are to be repipelined.

8.1 Pipeline Products

As with the EPIC and RGS, you will find a variety of OM-specific files in your data directories. The pipeline products differ slightly with different versions of the SAS software. We give a brief description of the files produced by SAS V9, and discuss the important differences with older pipeline products.

8.1.1 Imaging Mode

The PPS directory for the OM products contains files with nomenclature as described in Tables 3.2 and 3.3. As can be seen in those tables, the OM produces, among other things, sky images (*SIMAGE*.FTZ) and source lists (*SWSRLI*.FTZ). There is a low resolution sky image for each filter; they follow the nomenclature:

• PjjjjjjkkkkOMX000RSIMAGbb000.QQQ

  jjjjjj - Proposal number

  kkkk - Observation ID

  b - Filter keyword: B, V, U, L (UVW1) and S (UVW2)

  zzz - File type (e.g., PNG, FTZ)

For example, P0123456789OMX000RSIMAGB000.FTZ is the low-resolution final image in the B filter of the observation 0123456789 in sky coordinates (indicated by the S before the IMAG). The letter L is used for the UVW1 filter and S for UVW2. The keyword XPROC0 in the FITS header lists the files which have been added to create the final image P0123456789OMX000RSIMAGB000.FTZ. The keyword looks like this:

XPROC0  = 'ommosaic imagesets=''"product/P0123456789OMS008SIMAGE1000.FIT"&'
CONTINUE  ' "product/P0123456789OMS409SIMAGE1000.FIT" "product/P01234567&'
CONTINUE  '89OMS410SIMAGE1000.FIT" "product/P0123456789OMS411SIMAGE1000.&'
CONTINUE  'FIT" "product/P0123456789OMS412SIMAGE1000.FIT"'' mosaicedset=''&'
CONTINUE  'product/P0123456789OMX000RSIMAGS000.FIT'' sampling=''point'' # (&'
CONTINUE  'ommosaic-1.2.1) [xmmsas_20011206_1713-no-aka]'

The source list file (*SWSRLI*.FTZ) also contains useful information for the user; the column names are listed in Table 8.1.

Table 8.1: Some of the important columns in the SWSRLI FITS file.

Column name	Contents
SRCNUM	Source number
RA	RA of the detected source
DEC	Dec of the detected source
POSERR	Positional uncertainty
RATE	extracted count rate
RATE_ERR	error estimate on the count rate
SIGNIFICANCE	Significance of the detection (in $\sigma$)
MAG	Brightness of the source in magnitude
MAGERR	uncertainty on the magnitude

Figure 8.1: Merged OM image of the Lockman Hole SV1 observation obtained with the V filter. The image is displayed in logarithmic scale with an upper cut value of 20,000.

8.2 Rerunning the Pipeline

Throughout the OM section of this ABC Guide, public data from the Lockman Hole SV1 observation (Obs ID 0123700101) have been used to illustrate the SAS tasks. We suggest that the user download these data and retrace the following procedures. Figure 8.1 shows the merged V-band image from the Lockman Hole SV1 observation using the ommosaic task.

8.2.1 Prepare the Dataset

First, we must consider which files need to be processed. A good place to start is by grouping the ODF files by filter values. This can be done by using the Ftools task fkeyprint to print out the values for the header keyword FILTER:

fkeyprint odfile_name FILTER

The FILTER keyword in the initial ODF file is a number between 0 and 2100. The correspondence between number and filter value is given in Table 8.2.

Table 8.2: OM filter and file name correspondence.

File ID	Filter
1200	blocked
1400	V
1600	Magnifier
1800	U (no bar)
2000	B
0000	White (datum)
0200	Grism 2 (Optical)
0400	UVW1
0600	UVM2
0800	UVW2
1000	Grism 1 (UV)
2100	Bar

In general, the number following _OMS will either be of the form 00400, 00401, 00500... or 40100, 40101, 40200,.. The last two digits indicate the resolution. 00 is high-resolution and 01 is low-resolution. In this example, the high-resolution window will be called 0070_0123700101_OMS00400IMI.FIT.gz while the low-resolution window will be 0070_0123700101_OMS00401IMI.FIT.gz. The low-resolution images for each of the five frames are taken consecutively to obtain the full FOV. For each low-resolution frame there is a high-resolution frame of the inner part of the detector.

Please be aware that you should NOT add low-resolution and high resolution images, even if they cover the same part of the FOV (that is, you cannot add 0070_0123700101_OMS00401IMI.FIT and 0070_0123700101_OMS00400IMI.FIT).

As this example, we will use the first high-resolution exposure for the Lockman Hole SV1 data. Since we will need other files associated with this exposure for processing, we will copy them over to our usual reprocessing directory PROC:

cp 0070_0123700101_OMS00*FIT ../PROC

cp 0070_0123700101_OMX00000*.FIT ../PROC

cp 0070_0123700101_SCX0* ../PROC

There should be eight files in all:

/XMM/PROC: ls

$$ 0070_0123700101_OMS00400IMI.FIT 0070_0123700101_OMX00000PEH.FIT

$$ 0070_0123700101_OMS00400THX.FIT 0070_0123700101_SCX00000ATS.FIT*

$$ 0070_0123700101_OMS00400WDX.FIT 0070_0123700101_SCX00000SUM.ASC*

$$ 0070_0123700101_OMX00000NPH.FIT 0070_0123700101_SCX00000SUM.SAS

The file 0070_0123700101_SCX00000SUM.SAS has been edited to point to that directory, SAS_ODF is also pointing to this directory, and SAS_CCF points to the file ccf.cif generated by cifbuild (§ 5.1).

We must now prepare the dataset we are interested in by processing it with the task omprep. This task will need to be run twice: once with the THX file as input, and again with the IMI file as input.

To prepare the data with omprep from the command line:

1)

In the window where SAS was initialized, and in the directory where you want the output to go, type the following:

omprep set='0070_0123700101_OMS00400THX.FIT'

$$ pehset='0070_0123700101_OMX00000PEH.FIT'

$$ nphset='0070_0123700101_OMX00000NPH.FIT'

$$ wdxset='0070_0123700101_OMS00400WDX.FIT'

$$ outset='0070_0123700101_OMS00400THX_OUT_OMPREP.FIT'

$$ modeset=0

where

set - Tracking History Data Auxiliary file

pehset - Periodic Housekeeping file

nphset - Non-Periodic Housekeeping file

wdxset - Window Data Auxiliary file

outset - Output file

modeset - specifies if run in imaging mode (0), fast mode (1), slew mode (2), or tracking mode (3)

2)

And do the same for the IMI file:

omprep set='Mydata/0070_0123700101_OMS00400IMI.FIT'

$$ pehset='0070_0123700101_OMX00000PEH.FIT'

$$ nphset='0070_0123700101_OMX00000NPH.FIT'

$$ wdxset='0070_0123700101_OMS00400WDX.FIT'

$$ utset='0070_0123700101_OMS00400IMI_OUT_OMPREP.FIT'

$$ modeset=0

The output THX file is ready to be used by the rest of the SAS tasks.

8.2.2 Examine the Tracking History

There are two ways to examine the tracking history of the OM: by inspecting the postscript file output of the task omdrifthist, or by looking at the count rates of the guide stars with omthconv. This second task will produce a FITS file containing up to 10 columns with the guide stars' count rates.

To check the OM tracking using omdrifthist from the command line:

1)

In the window where SAS was initialized, and in the directory where you want the output to go, type the following:

omdrifthist set='0070_0123700101_OMS00400THX_OUT_OMPREP.FIT'

$$ plotfile='0070_0123700101_OMS00400THX_drift.ps'

$$ trackradius=0.5 hardcopy=yes pages='1 2'

where

set - THX file output from the omprep task

plotfile - name of output postscript file

trackradius - radius of pointing accuracy

hardcopy - produce hardcopy? yes/no

pages - Pages to plot (maximum pages produced is 2)

To check the OM tracking using omthconv from the command line:

1)

In the window where SAS was initialized, and in the directory where you want the output to go, type the following:

omthconv thxset='0070_0123700101_OMS00400THX_OUT_OMPREP.FIT'

$$ nphset='0070_0123700101_OMX00000NPH.FIT'

$$ outset='THX_trackingStar.FIT'

where

thxset - THX file output from the omprep task

nphset - name of Non Periodic Housekeeping file

outset - Output file

8.2.3 Detecting Bad Pixels

The task omcosflag looks at the (processed) OM tracking history and applies it to the map of bad pixels defined in the CCF. The resulting new bad pixel map is then used by the source detection algorithms. Bad pixels are set to 1, good pixels are set to 0.

To generate a bad pixel map on the command line:

1)

In the window where SAS was initialized, and in the directory where you want the output to go, type the following:

omcosflag samplefactor=1 timefactor=1

$$ set='0070_0123700101_OMS00400IMI_OUT_OMPREP.FIT'

$$ thxset='0070_0123700101_OMS00400THX_OUT_OMPREP.FIT'

where

samplefactor - Spatial oversampling factor (default 1)

timefactor - Temporal sampling factor (default 1)

thxset - Corrected THX file (output from the first omprep task)

set - Corrected IMI file (output from the second omprep task)

8.2.4 Generate the Flat Field

OM flat field generation is implemented in the omichain command, but there is no flat field generation in the OM pipeline. Instead, users can run the task omflatgen to produce a unit flatfield, followed by omflatfield, which creates a tracking-shifted flatfield and applies it to an OM Science Window (OSW) Image.

The omflatfield task creates two output files: one is the actual image and the other (specified by the output parameter ppsflatset) contains the tracking-shifted version of the omflatgen file.

To generate and apply a flat field from the command line:

1)

In the window where SAS was initialized, and in the directory where you want the output to go, type the following:

omflatgen outset='OUT_FLATGEN.FIT'

where

outset - Name of the output file

2)

The output from omflatgen is then used as input for omflatfield.

omflatfield samplefactor ='1'

$$ set='0070_0123700101_OMS00400IMI_OUT_OMPREP.FIT'

$$ thxset= '0070_0123700101_OMS00400THX_OUT_OMPREP.FIT'

$$ inorbitflatset='OUT_FLATGEN.FIT'

$$ tsflatset='0070_0123700101_OMS00400PPSFLATSET.FIT'

$$ outset='0070_0123700101_OMS00400IMI_OUT_FLATFIELD.FIT'

where

samplefactor Sampling factor (to be set to 1)

set - Corrected IMI file (output of the omcosflag task)

thxset - Corrected THX file (output of the omprep task)

inorbitflatset - Unit file (Output of the omflatgen task)

tsflatset - Output name for the tracking history flatfield

outset - Output name for the flat field image

8.2.5 Correct for Fixed-Pattern Noise

The task ommodmap corrects a given OM Science Window (OSW) image for ``modulo-8'' spatial fixed-pattern noise that results from the OM centroiding algorithm performed by the on-board electronics (see documentation at ``$SAS_PATH/doc/ommodmap/ommodmap.html'' for more details).
Note that the ommodmap task does not lose counts, it simply redistributes them.

To correct for fixed-pattern noise from the command line:

1)

In the window where SAS was initialized, and in the directory where you want the output to go, type the following:

ommodmap set='0070_0123700101_OMS00400IMI_OUT_FLATFIELD.FIT'

$$ mod8set='0070_0123700101_OMS00400PPSMODE8SET_OUT.FIT'

$$ outset='0070_0123700101_OMS00400OUT_OMMODMAP.FIT'

$$ nsig=3 nbox=16 mod8product=yes

where

set - Input file (output of omflatfield)

mod8product - Produce a Pipeline Processing System (PPS) file?

mod8set - Name of the output modulo-8 tile

outset - Name of the corrected image

nsig - Significance level for sigma clipping

nbox - Size of the sliding box in units of 8 pixels

8.2.6 Perform Source Detection

The task omdetect uses two different algorithms for detecting point and extended sources; they are discussed in detail at http://xmm.esac.esa.int/sas/9.0.0/doc/omdetect/index.html.

The task has a lot of parameters (see below) but only set and outset are mandatory.

To detect sources with omdetect from the command line:

1)

In the window where SAS was initialized, and in the directory where you want the output to go, type the following:

omdetect set='0070_0123700101_OMS00400OUT_OMMODMAP.FIT'

$$ outset='0070_0123700101_OMS00400IMI_OUT_OMDETECT.FIT'

$$ levelimage='0070_0123700101_OMS00400LEVELIMAGE.FIT'

$$ regionfile='0070_0123700101_OMS00400oswList.reg'

$$ wdxset='0070_0123700101_OMS42200WDX.FIT'

$$ backgroundimage='0070_0123700101_OMS00400BKGIMAGE.FIT'

$$ minsignificance=0 detectextended=no nsigma=6

where

set - Input file (output of ommodmap)

outset - Name of the output source list file

nsigma - Number of $\sigma$ above background for a detection

wdxset - name of WDX file (needed for images produced before SAS V5.4)

backgroundimage - Name of output background image file

levelimage - Name of image of island detections

minsignificance - minimum significance of sources to be included in OM OSW file. If 0, it is not used

regionfile - Name of saoimage region file

detectextended - use algorithm for detecting extended sources? yes/no

8.2.7 Convert Source Counts to Magnitudes

The task ommag converts the list of given source counts to magnitudes in the appropriate instrumental band passes.

To compute instrumental magnitudes with ommag from the command line:

1)

In the window where SAS was initialized, and in the directory where you want the output to go, type the following:

ommag set='0070_0123700101_OMS00400IMI_OUT_OMDETECT.FIT'

where

set - OM OSW source list file name (output of omdetect)

There is a ``recipe'' to convert the UV count rates to flux. It was provided by Alice Breeveld (MSSL) and can be accessed at:
http://xmm.esac.esa.int/sas/7.0.0/watchout/Evergreen_tips_and_tricks/uvflux.shtml

8.2.8 Convert Source OM Positions to Sky Coordinates

The task omatt converts an OM OSW source list from pixels to sky coordinates. These sky coordinates are then used to produce a sky coordinate image.

To compute sky coordinates with omattfrom the command line:

1)

In the window where SAS was initialized, and in the directory where you want the output to go, type the following:

omatt set='0070_0123700101_OMS00400OUT_OMMODMAP.FIT'

$$ sourcelistset='0070_0123700101_OMS00400IMI_OUT_OMDETECT.FIT'

$$ ppsoswset='0070_0123700101_OMS00400FINAL_IMAGE.FIT'

$$ usecat=no catfile='' maxradecerr=1.0 maxrmsres=1.5 rotateimage=no

where

set - Input file (output of the ommodmap task)

sourcelistset - Source list (output of omdetect task)

ppsoswset - Output name for the corrected sky image

usecat - Do you want to use the USNO-SA 1 catalog?

catfile - Name of the USNO star catalog (default: 'usnocat.fit')

maxradecerr - Maximum allowed RA/dec error in astrometry fit

maxrmsres - Maximum allowed rms residual in astrometry fit

rotateimage - create rotated sky image?

Due to the large size of the catalog, it is not distributed. Users, however, can provide their own catalog if they wish. The format is that used for the USNO cross-correlation FITS products. In general, the usecat keyword should be set to no.

The pointing stability about the spacecraft boresight position is better than 1 $^{\prime\prime}$ (look at the tracking plots derived at the beginning). There is still a scatter of about 4 $^{\prime\prime}$ between the planned and actual pointing position.

8.2.9 Fast Mode

SAS has a working fast mode pipeline. If the data have not been processed by the latest version of SAS, the task omfchain should be run.

The chain works similarly to the imaging chain explained above, and consists of a Perl script which calls all the necessary tasks sequentially. It produces images of the detected sources, extracts events related to the sources and the background, and extracts the corresponding light curves. A more detailed description of the chain can be found in the SAS on-line help available at http://xmm.esac.esa.int/sas/current/doc/index.html. A summary of the task is shown in Figure 8.2.

Figure 8.2: OM fast chain-diagram of the different tasks run.

8.2.10 Grism Analysis

The metatask omgchain, can be used to extract and automatically calibrate spectra produced by the OM grisms. OM grism data are taken in Image Mode. Hence omgchain uses already existing tasks, such as omprep and ommodmap, to handle housekeeping information and to perform some corrections (the ``modulo-8'' noise reduction for example). Also, omdetect is designed to find the spectra, zero and first orders, producing a source list. Other tasks are grism specific. omgprep is used to correct for geometric distortion of the detector and to rotate the image so as to have the dispersion direction aligned with the image Y axis. omgprep performs the spectral extraction and the wavelength and flux calibration. Finally, the extracted spectra are plotted using omgplot.

The sequence of tasks used by omgchain is illustrated in Fig. 8.3. An output spectrum produced by omgchain is given in Fig. 8.4. Each of these tasks can be run individually. SAS V7 also includes a new interactive task, omgsource, which allows the user to select with the cursor the spectrum to be extracted.

Figure 8.3: Diagram of the different tasks used by omgchain. The first four tasks are preparatory, the other three tasks execute the source detection and spectral identification procedure, and produce the output files.

The task omgchain has many parameters, but none of them are mandatory. Omgchain will search for grism images in the working directory; if at least one such image is found, it will process it. If none are found, the task will produce no output.

Below is a description of the calling sequence and the individual parameters.

omgchain inpdirectory=../ODF outdirectory=PROC comment=''

$$nsigma=3 combine=yes spectrumhalfwidth=-8 bkgoffsetleft=0 bkgwidthleft=-8

$$bkgoffsetright=0 bkgwidthright=-8 spectrumsmoothlength=0 mod8correction=1

$$extractionmode=0 plotbinsize=1 plotflux=2 scalebkgplot=no

where

inpdirectory - Input file directory

outdirectory - Output file directory

comment - User's comments for output

nsigma - Number of $\sigma$ above the background required for a detection (this parameter is passed to omdetect

combine - Condition for combining the Engineering-2 subwindows

spectrumhalfwidth - Half-width of the spectrum extraction region in pixels, if negative, and in FWHMs otherwise

bkgoffsetleft - Offset of the left background extraction region from the edge of the spectrum extraction area; in pixels, if negative, or in FWHMs otherwise.

bkgwidthleft - Width of the left background extraction region; in pixels, if negative, or in FWHMs otherwise

bkgoffsetright - Offset for the right background extraction region; in pixels, if negative, and in FWHMs otherwise

bkgwidthright - Width of the right background extraction region; in pixels, if negative, or in FWHMs otherwise.

spectrumsmoothlength - Length of the smoothing window for smoothing the extracted spectra, if necessary. Values 0 or 1 of this parameter imply no smoothing

mod8correction - Condition for removing the modulo-8 noise: 0: correction not applied; 1: correction applied using the modulo-8 map extracted from the input image; 2: correction applied using the modulo-8 map extracted from the OM CCF flat field; 3: correction applied multiplying the input image by the OM CCF flat field

extractionmode - Switch between different extraction modes. The value 0 corresponds to the normal extraction (summation of counts in the cross-dispersion direction); 1 corresponds to the Optimal Extraction; 2 corresponds to the spline smoothing; 3 corresponds to the Gaussian fit

extractfieldspectra - Condition for extraction either only the target object spectrum or all available spectra of the sources in the field

plotbinsize - Size of spectrum wavelength bins for the output plot (in Å)

plotflux - Flag for plotting the spectrum only (value 0), the background only (value 1), or both of them (value 2)

scalebkgplot - Condition for scaling the background plot differently from the spectrum plot

If if a source is not detected by omdetect, or does not fall within the grism window, omgchain will run nonetheless without giving any warnings, but will not produce output files.

Figure 8.4: OM optical grism spectrum obtained from a 4.7 ks observation of Mrk 478.

8.3 In a Nutshell

To summarize, the following steps are needed to process OM data:

1) Obtain the raw and pipeline data.

2) Initialize SAS.

3) Make the CCF and ODF summary file (run the cifbuild and odfingest tasks).

4) Rerun the appropriate pipline chain (omichain, omgchain, or omfchain). The basic steps of omichain have been described above.