Next: 8. EPIC-PN Data Processing Up: XMM ABC Guide Previous: 6. Preparing the Data Contents
Upon receipt of an XMM-Newton EPIC data set, it is recommended
to check what the observation consists of (§ 3.2) and note
when the observation was taken. If it is a recent observation (within the
last year), it was likely processed with the most recent calibrations and
SAS, and you can immediately start to analyze the Pipeline Processed data
(though reprocessing it with the current SAS will do no harm.) However, if
it is more than a year old, it was probably processed with older versions
of CCF and SAS prior to archiving, and the pipeline should be rerun to
generate event files with the latest calibrations.
As noted in
Chapter 5, a variety of analysis packages can be used for the
following steps. However, as the SAS was designed for the basic reduction
and analysis of XMM-Newton data (extraction of spatial, spectral, and
temporal data), it will be used here for demonstration purposes.
SAS will be required at any rate for the production of detector response
files (RMFs and ARFs) and other observatory-specific requirements,
though for the simple case of on-axis point sources the canned
response files provided by the SOC can be used.
NOTE: For PN observations with very bright sources, out-of-time
(OoT) events can provide a serious contamination of the image. They are
immediately recognizable as a bright stripe in the image between the
source and the edge of the chip. They occur because the readout period
for the CCDs can be up to 
% of the frame time. Since events
that occur during the read-out period can't be distinguished from other
events, they are included in the event files but have invalid locations.
It is strongly recommended that you keep all reprocessed data in its own
directory! SAS places output files in whichever directory it is in when a task
is called. Throughout this primer, it is assumed that the Pipleline Processed data are
in the PPS directory, the ODF data (with upper case file names, and uncompressed)
are in the ODF directory, the analysis is taking place in the PROC directory, and
the CCF data are in the directory CCF.
If your data are recent, you need only to gunzip the files and prepare the data for processing (§6). Feel free to skip the section on repipelining (§7.1 and proceed to later discussions. In any case, for simplicity, it is recommended that you change the name of the unzipped event file to something easy to type. For example, an MOS1 event list:
| cp PPS/PiiiiiijjkkM1SlllMIEVLI0000.FIT PROC/m1.fits |
where
|
Various analysis procedures are demonstrated in this chapter using the Lockman Hole SV1 dataset,
ObsID 0123700101. The following procedures
are applicable to all XMM-Newton datasets, so it is not required that you use this
particular dataset; any observation should be sufficient.
If you simply want to have a quick look at your data, the ESKYIM files contain
EPIC sky images in different energy bands whose ranges are listed in
Table 3.3. While the zipped FITS files may need to be unzipped
before display in ds9 (depending on the version of ds9), they can be
displayed when zipped using fv (fv is FITS file viewer available in the
HEASoft package). In addition, the image of the total band pass for all three EPIC
detectors is also provided in PNG format which can be displayed with a web browser.
Also, the PP source list is provided in both zipped FITS format (readable by
fv) and as an HTML file.
For detailed descriptions of PP data nomenclature, file contents, and
which tasks can be used to view them, see Tables 3.2 and 3.3.
For detailed descriptions of ODF data nomenclature and file contents, see Table 3.1.
We assume that the data was prepared and environment variables were set
according to §6. There are a couple ways to do this;
we can use “the procs” (i.e., emproc and epproc) or
“the chains” emchain and epchain; the outcome is the
same for all intents and purposes, so the choice is left to the user. Both
are shown below.
Emproc and epproc will automatically detect
if the data were taken in imaging or timing mode. If you are using data that
were not taken in Imaging mode and want to run emchain or
epchain, you will need to set the relevant parameter, as shown
below.
To process the MOS and PN data with the procs, type
| emproc |
and
| epproc |
To do this from the GUI,
| 1) | Call emproc. |
| 2) | Click “Run”. |
and
| 1) | Call epproc. |
| 2) | Click “Run”. |
If we wanted to use the chains instead, we would need to call HEASoft's “ftools” first:
| ftools |
And then call the chains:
| emchain |
and
| epchain |
At this time, the chains cannot be called from the GUI.
There are parameters that we can edit to suit our needs. For example, to create an out-of-time event file for the PN data, add the parameter withoutoftime to your invocation:
| epproc withoutoftime=yes |
or
| epchain withoutoftime=yes |
To use the GUI,
| 1) | Call epproc. |
| 2) | In the “Frames and Events” tab, toggle withoutoftime to yes. |
| 3) | Click “Run”. |
To process data from only MOS1, not MOS2, enter
| emproc selectinstruments=yes emos1=yes |
or
| emchain instruments=M1 |
To use the GUI,
| 1) | Call emproc. |
| 2) | In the “Data Processing” tab, check the selectinstruments box, then toggle emos1 to yes. |
| 3) | Click “Run”. |
By default, these tasks do not keep any intermediate files they generate.
The final files are numerous, but the only ones we are interested in are
the event files. Emproc and epproc designate their output
event files with the suffix “ImagingEvts.ds”. Emchain and
epchain maintain the naming
convention described in §3.2.4; in short, the event files have
“EVLI” in their names. In any case, it is convenient to use names
that are easy to type, so we will make soft links to new names.
Before we do so, however, we note that there are two MOS1 exposures, one of which
has an “S”, for “scheduled”, and the other with “U”, for “unscheduled”, in its exposure ID
(0070_0123700101_EMOS1_S001_ImagingEvts.ds and
0070_0123700101_EMOS1_U002_ImagingEvts.ds;
P0123700101M1S001MIEVLI0000.FIT and
P0123700101M1U002MIEVLI0000.FIT).
These indicate that the exposure was interrupted, with the scheduled observation
being the part that was planned, and the unscheduled one being the rest of the
observation after it resumed.
In cases where an observation was interrupted, it is possible that the resulting event files are too short to be useful. Let's see how long the MOS1 observations are. We can do this a couple different ways. First, we could use ftools; if you have not called ftools yet since you did not run emchain and epchain, you will need to call it now, though there is no harm in calling it again. Please note that you may need to insert a slash (“\”) before the opening and closing square brackets in the fkeyprint command.
| ftools
fkeyprint 0070_0123700101_EMOS1_S001_ImagingEvts.ds[1] ONTIME fkeyprint 0070_0123700101_EMOS1_U002_ImagingEvts.ds[1] ONTIME |
Alternatively, we could call up “ds9” and view the headers with that by
going to the menu bar and selecting File 
Header and m1.fits[EVENTS]
in the pop-up window.
| ds9 0070_0123700101_EMOS1_S001_ImagingEvts.ds &
ds9 0070_0123700101_EMOS1_U002_ImagingEvts.ds & |
Either way, we see that the scheduled observation is about 4x longer than the
unscheduled one. For the remainder for this example case, we will opt to ignore
the unscheduled observation and consider only the scheduled one.
To make soft links to the event files made with emproc and epproc, type
| ln -s 0070_0123700101_EMOS1_S001_ImagingEvts.ds m1.fits
ln -s 0070_0123700101_EMOS2_S002_ImagingEvts.ds m2.fits ln -s 0070_0123700101_EPN_S003_ImagingEvts.ds pn.fits |
To do so for the output from emchain and epchain, type
| ln -s P0123700101M1S001MIEVLI0000.FIT m1.fits
ln -s P0123700101M2S002MIEVLI0000.FIT m2.fits ln -s P0123700101PNS003PIEVLI0000.FIT pn.fits |
If we take a look at our event lists in ds9, we can see that they are noisy. Luckily, it is quite straightforward to filter them. The filtering expressions for the MOS and PN are, respectively:
(PATTERN  12) && (PI in [200:12000]) && #XMMEA_EM
|
and
| (PATTERN 4) && (PI in [200:15000]) && #XMMEA_EP
|
In both, the first expression will select good events with PATTERN
in the 0 to 12 range for MOS and 0 to 4 range for PN. Event patterns are
discussed in §4, but to summarize here, it is related
to the number and pattern of the CCD pixels triggered for a given event. The
PATTERN assignments are: single pixel events: PATTERN == 0,
double pixel events: PATTERN in [1:4], triple and quadruple events:
PATTERN in [5:12].
The second keyword in the expressions, PI, selects the pulse
height of the event and is related to PHA; for the MOS, this should be
between 200 and 12000 eV. For the PN, this should be between 200 and 15000 eV.
This should clean up the event lists significantly, with most of the rest of
the obvious contamination due to low pulse height events. Setting the
lower PI channel limit somewhat higher (e.g., to 300 or 400 eV) will eliminate
much of the rest.
Finally, the #XMMEA_EM (#XMMEA_EP for the PN) filter provides
a canned screening set of FLAG values for the event. The FLAG value provides a
bit encoding of various event conditions, e.g., near hot pixels or outside of the
field of view. Setting FLAG == 0 in the selection expression provides the
most conservative screening criteria and should always be used when serious spectral
analysis is to be done on the PN. It typically is not necessary for the MOS.
It is a good idea to keep the output filtered event files and use them in your
analyses, as opposed to re-filtering the original file with every task. This will
save much time and computer memory. As an example, the Lockman Hole data's original
MOS2 event file is 36 MB; the fully filtered list (that is, filtered spatially, temporally,
and spectrally) is only 3 MB!
Lastly, we have reached the point in this walk-through where our commands
get long. As mentioned in §6.2, we will start to use
a “\” to indicate continuation on the next line of text.
To apply the standard filters, type:
evselect table=m1.fits filteredset=m1_filt.fits \
 expression='(PATTERN 12) && (PI in [200:12000]) && #XMMEA_EM'
|
| evselect table=m2.fits filteredset=m2_filt.fits \
expression='(PATTERN 12) && (PI in [200:12000]) && #XMMEA_EM'
|
| evselect table=pn.fits filteredset=pn_filt.fits \
expression='(PATTERN 4) && (PI in [200:15000]) && #XMMEA_EP'
|
where
| table - input event table
expression - filtering expression filteredset - output file name |
To do this with the GUI, for MOS1,
| 1) | Call evselect. |
| 2) | In the “General” tab, in table, enter the event file, m1.fits.
Check the “keepfilteroutput” and “withfilteredset” boxes, then set
filteredset to the output file name, m1_filt.fits. In the
expression box, enter the filtering expression,
(PATTERN 12) && (PI in [200:12000]) && #XMMEA_EM.
|
| 3) | Click “Run”. |
Repeat these steps with the appropriate changes for MOS2 and PN.
Sometimes, it is necessary to use filters on time in addition to those
mentioned above. This is because of soft proton background flaring, which
can have count rates of 100 counts/sec or higher across the entire bandpass.
More information on soft proton flares can be found in §4.2.
To determine if our observation is affected by background flaring, we can examine the
high-energy, single-pattern event light curve. Note that for PN, we cap the high energy
limit at 12 keV; this is to prevent the misidentification of hot pixels as flares.
First, let's prepare a directory to keep things organized:
| mkdir ltcrv
cd ltcrv ln -s ../m1_filt.fits ln -s ../m2_filt.fits ln -s ../pn_filt.fits |
Now we will make our light curves. For the time binning, we will set it to something reasonable (usually between 10 and 100 s).
| evselect table=m1_filt.fits withrateset=yes rateset=m1_ltcrv.fits \
maketimecolumn=yes timebinsize=100 makeratecolumn=yes \
expression='(PATTERN==0) && (PI > 10000)'
evselect table=m2_filt.fits withrateset=yes rateset=m2_ltcrv.fits \
evselect table=pn_filt.fits withrateset=yes rateset=pn_ltcrv.fits \
|
where parameters are as defined above, and
| withrateset - make a light curve
rateset - name of output light curve file maketimecolumn - control to create a time column timecolumn - time column label timebinsize - time binning (seconds) makeratecolumn - control to create a count rate column, otherwise a count column will be created |
To do this with the GUI, for MOS1,
| 1) | Call evselect. |
| 2) | In the “General” tab, in table, enter the event file, m1_filt.fits. In the expression box, enter the filtering expression, (PATTERN==0) && (PI > 10000). In the “Lightcurve” tab, check the withrateset box. In the rateset box, enter the name of the output lightcurve, m1_ltcrv.fits. In the timebinsize box, enter the bin size in seconds. Check the maketimecolumn and makeratecolumn boxes. |
| 3) | Click “Run”. |
Repeat these steps with the appropriate changes for MOS2 and PN.
The output files can be viewed by using the ftool fv. If you have not called ftools yet, remember to do so before calling fv. (There is no harm in calling it more than once.) To see the MOS1 light curve, type
| fv m1_ltcrv.fits & |
In the pop-up window, the RATE extension will be available in the second row (index 1, as numbering begins with 0). Select “PLOT” from this row, and select the column name and axis on which to plot it. We can open the other light curves and view them in the same window; see Figure 7.1.
Taking a look at the light curve, we can see that there is a very large
flare toward the end of the observation and three much smaller ones in the
middle of the exposure.
There are many ways to filter on time: by making a secondary Good Time Interval (GTI) file with the task tabgtigen, which will allow us to filter on time or rate, or by making a new GTI file with the task gtibuild using time as the filter, or with an explicit reference to the time in the evselect filtering expression. Any of these methods will produce a cleaned file, so which one to use is a matter of the user's preference and what makes the most sense for the science being done. For the example below, we will use the PN light curve, though a MOS light curve would also work.
Filter on RATE with tabgtigen
Examining the PN light curve shows us that during non-flare times, the count rate is very low, about 0.1 c/s. It looks like setting a cap of 0.2 c/s should remove most of the flares. We can use that to generate the secondary GTI file:
| tabgtigen table=pn_ltcrv.fits gtiset=gti.fits expression='RATE 0.2'
|
where parameters are as defined above, and
| gtiset - output file name for selected GTI intervals
|
To do this with the GUI,
| 1) | Call tabgtigen. |
| 2) | In table, select the event file, m1_filt.fits.
In the expression box, enter the filtering expression,
RATE 0.2. In the gtiset box, enter the output file name,
gti.fits.
|
| 3) | Click “Run”. |
We can use evselect to apply it:
| evselect table=m1_filt.fits filteredset=m1_filt_time.fits expression='GTI(gti.fits,TIME)'
evselect table=m2_filt.fits filteredset=m2_filt_time.fits expression='GTI(gti.fits,TIME)' evselect table=pn_filt.fits filteredset=pn_filt_time.fits expression='GTI(gti.fits,TIME)' |
where the parameters are as defined in §7.2.
To do this with the GUI, for MOS1,
| 1) | Call evselect. |
| 2) | In the “General” tab, in table, enter the event file, m1_filt.fits. Check the “keepfilteroutput” and “withfilteredset” boxes, then set filteredset to the output file name, m1_filt_time.fits. In the expression box, enter the filtering expression, GTI(gti.fits,TIME). |
| 3) | Click “Run”. |
Repeat these steps with the appropriate changes for MOS2 and PN.
Make a Filter on TIME with tabgtigen
Alternatively, we could have chosen to make a new GTI file by noting the times
of the flaring in the light curve and using that as a filtering parameter. The
big flare starts around 7.32276e7 s, and the smaller ones are at 7.32119e7 s,
7.32204e7 s, and 7.32229e7 s. The expression to remove these would be (TIME
7.32273e7) &&! (TIME IN [7.32117e7:7.32121e7]) &&! (TIME IN [7.32202e7:7.32206e7]
&&! (TIME IN [7.32219e7:7.32226).
The syntax (TIME 7.32273e7) includes only events with times less
than 7.32273e7 s, and the “!” symbol stands for “not”, so the time ranges that
follow &&! are excluded. Once the new GTI file is made, we apply it with
evselect in the usual way.
| tabgtigen table=pn_ltcrv.fits gtiset=gti.fits expression= \
'(TIME 7.32273e7) &&! (TIME IN [7.32118e7:7.32121e7]) \
&&! (TIME IN [7.32202e7:7.32206e7] &&! (TIME IN [7.32219e7:7.32226)'
evselect table=m1_filt.fits filteredset=m1_filt_time.fits expression='GTI(gti.fits,TIME)'
|
where the parameters are as defined above.
To do this with the GUI,
| 1) | Call tabgtigen. |
| 2) | In table, select the event file pn_filt.fits.
In the expression box, enter the filtering expression,
(TIME 7.32273e7) &&! (TIME IN [7.32118e7:7.32121e7]) &&!
(TIME IN [7.32202e7:7.32206e7] &&! (TIME IN [7.32219e7:7.32226). In the gtiset box,
enter the output file name, gti.fits.
|
| 3) | Click “Run”. |
| 4) | Call evselect. |
| 5) | In the “General” tab, in table, enter the event file, m1_filt.fits. Check the “keepfilteroutput” and “withfilteredset” boxes, then set filteredset to the output file name, m1_filt_time.fits. In the expression box, enter the filtering expression, GTI(gti.fits,TIME). |
| 6) | Click “Run”. |
Repeat step 5 with the appropriate changes for MOS2 and PN.
Make a Filter on TIME with gtibuild
This method requires a text file as input. In the first two columns, we enter the start and end times (in seconds) that we are interested in, and in the third column, indicate with either a + or - sign whether that region should be kept or removed. Each good (or bad) time interval should get its own line, with any optional comments preceeded by a “#”. In the example case, we would write in our ASCII file (named gti.txt):
| 0 7.32273e7 + # Good time in the first 2/3 of the observation
7.32117e7 7.32121e7 - # but without a small flare here, 7.32202e7 7.32206e7 - # and here, 7.32219e7 7.32226 - # and here. |
and proceed to gtibuild:
| gtibuild file=gti.txt table=gti.fits |
where
| file - input text file
table - output GTI file |
To do this with the GUI,
| 1) | Call gtibuild. |
| 2) | In file, enter the name of the text file, gti.txt. In the table box, enter the output file name, gti.fits. |
| 3) | Click “Run”. |
And we apply it in the usual manner:
| evselect table=m1_filt.fits filteredset=m1_filt_time.fits expression='GTI(gti.fits,TIME)'
evselect table=m2_filt.fits filteredset=m2_filt_time.fits expression='GTI(gti.fits,TIME)' evselect table=pn_filt.fits filteredset=pn_filt_time.fits expression='GTI(gti.fits,TIME)' |
where the parameters are as described previously.
To do this with the GUI, for MOS1,
| 1) | Call evselect. |
| 2) | In the “General” tab, in the table box, enter the event file, m1_filt.fits. Check the “keepfilteroutput” and “withfilteredset” boxes, then set filteredset to the output file name, m1_filt_time.fits. In the expression box, enter the filtering expression, GTI(gti.fits,TIME). |
| 3) | Click “Run”. |
Repeat these steps with the appropriate changes for MOS2 and PN.
Filter on TIME by Explicit Reference
Finally, we could have chosen to forgo making a secondary GTI file altogether, and simply filtered on time in a manner similar to applying the standard filtering expression (see §7.2):
| evselect table=m1_filt.fits filteredset=m1_filt_time.fits expression= \
'(TIME 7.32273e7) &&! (TIME IN [7.32117e7:7.32121e7]) &&! \
(TIME IN [7.32202e7:7.32206e7]) &&! (TIME IN [7.32219e7:7.32226])'
evselect table=m2_filt.fits filteredset=m2_filt_time.fits expression= \
evselect table=pn_filt.fits filteredset=pn_filt_time.fits expression= \
|
where the keywords are as described previously.
To do this with the GUI, for MOS1,
| 1) | Call evselect. |
| 2) | In the “General” tab, in table, enter the event file, m1_filt.fits.
Check the “keepfilteroutput” and “withfilteredset” boxes, then set
filteredset to the output file name, m1_filt_time.fits. In the
expression box, enter the filtering expression,
(TIME 7.32273e7) &&! (TIME IN [7.32117e7:7.32121e7]) &&!
(TIME IN [7.32202e7:7.32206e7]) &&!
(TIME IN [7.32219e7:7.32226]). |
| 3) | Click “Run”. |
Repeat these steps with the appropriate changes for MOS2 and PN.
Now that we have made and applied the time filter, we will take another look
at our data in ds9. As seen in Figure 7.2, the
MOS data appear clean, and we can use the filtered files in all the imaging
and spectral analysis tasks below. However, we can see that there is still a
bit of noise in the PN. This is due to our pattern selection in
§7.2, which included both single- and double-pattern
events: double-pattern events are only created above 2x the low-energy threshold,
which means that the cleanest images are produced by excluding the energy range
just above the threshold.
In practical terms, this means that if we want to make a PN image from 0.2
to 15 keV, our options are to either
1) use only the single-pattern events for 
keV, and singles+doubles
for 
keV; or
2) set FLAG==0, which will remove areas of the detector like border
pixels and columns with higher offsets but also result in more stringent
filtering than needed at higher energies; or
3) revisit our decision to use the 0.2-0.4 keV band in PN.
Which approach to take will depend on what further anaylsis we wish to do.
Since we will be doing source detection at higher energies and these are
faint sources, we'll opt to use only 
0.4 keV. (We will also extract
spectra, but we can apply filters as needed when the time comes.) An
image of the PN data from 0.4-15 keV is shown in the last panel of
Figure 7.4.
At this point, the data are clean and are ready for analysis. Some useful SAS tasks for source detection, making an exposure map, and extracting a spectrum are shown below.
The edetect_chain task does nearly all the work involved with EPIC source
detection. It can process up to three intruments (both MOS cameras and the PN) with up
to six images in different energy bands simultaneously. All images must have identical
binning and WCS keywords. We will do a simultaneous source detection in the three EPIC cameras,
in two bands that correspond to bands 2 (0.5-1 keV) and 5 (4.5-12 keV) in the XMM
Serendipitous Source catalog.
This dataset was taken using the THIN filter, so we will use the energy conversion factors
(ECFs), which convert band count rates to fluxes, associated with that filter.
In the filtering expressions below, we use the “
” symbols to indicate “or”.
Note that edetect_chain requires ftools to run, so if we had not
called ftools before, we'd need to do so now.
Users interested in other bands might wish to look at
Table 1 in the 4XMM Catalogue User Guide;
the energy conversion factors are listed in
Table 8 of the 3XMM-DR6 Catalogue User Guide.
We will start by generating the files that edetect_chain needs: an attitude file, and images in each detector in the energy bands of interest. First, let's prepare a directory to keep things organized:
| cd ..
mkdir source_detection cd source_detection ln -s ../ltcrv/m1_filt_time.fits ln -s ../ltcrv/m2_filt_time.fits ln -s ../ltcrv/pn_filt_time.fits |
We can make the attitude file by typing
| atthkgen atthkset=attitude.fits |
where
| atthkset - output file name
|
To do this with the GUI,
| 1) | Call atthkgen. |
| 2) | Set the atthkset keyword to the output file name, attitude.fits. |
| 3) | Click “Run”. |
Next, make the soft and hard X-ray images:
| evselect table=m1_filt_time.fits withimageset=yes imageset=m1-s.fits \
imagebinning=binSize xcolumn=X ximagebinsize=40 ycolumn=Y yimagebinsize=40 \
expression='PI in [500:1000]'
evselect table=m2_filt_time.fits withimageset=yes imageset=m2-s.fits \
evselect table=pn_filt_time.fits withimageset=yes imageset=pn-s.fits \
evselect table=m1_filt_time.fits withimageset=yes imageset=m1-h.fits \
evselect table=m2_filt_time.fits withimageset=yes imageset=m2-h.fits \
evselect table=pn_filt_time.fits withimageset=yes imageset=pn-h.fits \
|
We will also make an image for each detector with both soft and hard X-rays for display purposes:
| evselect table=m1_filt_time.fits withimageset=yes imageset=mos1-all.fits \
imagebinning=binSize xcolumn=X ximagebinsize=40 ycolumn=Y yimagebinsize=40 \
expression='(PI in [500:1000]) || (PI in [4500:12000])'
evselect table=m2_filt_time.fits withimageset=yes imageset=m2-all.fits \
evselect table=pn_filt_time.fits withimageset=yes imageset=pn-all.fits \
|
where the parameters are as defined above and
| withimageset - toggle to create an image
imageset - fits image name to be created imagebinning - how to bin the image xcolumn - table column to use for the X axis ximagebinsize - binning in X axis ycolumn - table column to use for the Y axis yimagebinsize - binning in Y axis |
To do this with the GUI, starting with the MOS1 soft image,
| 1) | Call evselect. |
| 2) | In the “General” tab, in table, enter the event file, m1_filt_time.fits. In the expression box, enter the filtering expression, PI in [500:1000]. In the “Image” tab, check the withimageset box and in the imageset box, enter the output image name, m1-s.fits. Set xcolumn to X and ycolumn to Y. Set Binning to binSize, and both ximagebinsize and yimagebinsize to 40. |
| 3) | Click “Run”. |
Repeat these steps with the appropriate changes for MOS2 and PN, then do so again using the
different energy bands.
Now we can run edetect_chain:
| edetect_chain imagesets='m1-s.fits m1-h.fits m2-s.fits m2-h.fits pn-s.fits pn-h.fits' \
eventsets='m1_filt_time.fits m2_filt_time.fits pn_filt_time.fits' \
attitudeset=attitude.fits \
pimin='500 4500 500 4500 500 4500' pimax='1000 12000 1000 12000 1000 12000' \
ecf='1.7461 0.1450 1.7572 0.1528 8.1210 0.5764' eboxl_list=all_eboxlist_l.fits \
eboxm_list=all_eboxlist_m.fits eml_list=all_emllist.fits
|
where
| imagesets - list of count images
eventsets - list of event files attitudeset - attitude file name pimin - list of minimum PI channels for the bands pimax - list of maximum PI channels for the bands ecf - energy conversion factors for the bands eboxl_list - output file name for the local sliding box source detection list
eboxm_list - output file name for the sliding box source detection in background map mode list
eml_list - output file name for maximum likelihood source detection list |
To do this with the GUI,
| 1) | Call edetect_chain. |
| 2) | In the “0” tab, in the imagesets box, enter the images: m1-s.fits m1-h.fits m2-s.fits m2-h.fits pn-s.fits pn-h.fits. In the eventsets box, enter the event files: m1_filt_time.fits m2_filt_time.fits pn_filt_time.fits. In the attitudeset box, enter the attitude file, attitude.fits. Set the pimin keyword to the minimum PI values (in eV) for the input images: 500 4500 500 4500 500 4500 and do the same for the maximum values for pimax: 1000 12000 1000 12000 1000 12000. Set ecf to 1.7461 0.1450 1.7572 0.1528 8.1210 0.5764. |
| 3) | In the “1” tab, set eboxl_list to all_eboxlist_l.fits and eboxm_list to all_eboxlist_m.fits. |
| 4) | In the “2” tab, set eml_list to all_emllist.fits. |
| 5) | Click “Run”. |
We will make a mosaicked image for display purposes:
| emosaic imagesets='m1-all.fits m2-all.fits pn-all.fits' mosaicedset=mosaic.fits |
where
| imagesets - images to be mosaicked
mosaicedset - output file |
To do this with the GUI,
| 1) | Call emosaic. |
| 2) | In the imagesets box, enter the images: m1-all.fits m2-all.fits pn-all.fits. In the mosaicedset box, enter the output name, mosaic.fits. |
| 3) | Click “Run”. |
We can use srcdisplay to view the mosaic and make a region file that we can use with ds9. Note that, while srcdisplay will send an error message to the screen saying that it cannot lauch ds9, it actually does open a ds9 window that can be used to view our mosaicked image and regions. The source region radius is in units of degrees, and the default value, 0.01 degrees, works well here. However, we can easily change it if we want to with the sourceradius parameter (e.g., sourceradius=0.05).
| srcdisplay boxlistset=all_emllist.fits imageset=mosaic.fits
regionfile=sources.reg withregionfile=yes
|
where
| boxlistset - eboxdetect source list
imageset - image file on which the source circles are to be plotted regionfile - file name of output file containing source regions withregionfile - flag to create a region file |
To do this with the GUI,
| 1) | Call srcdisplay. |
| 2) | In the boxlistset box, enter the source list, all_emllist.fits. In the imageset box, enter mosaic.fits. Check the box next to withregionfile, then in the regionfilebox, enter the name of the output region file, sources.reg. |
| 3) | Click “Run”. |
Figure 7.3 shows the mosaic overlayed with the detected sources.
Exposure maps are images that contains the spatial efficiency of an instrument.
They are used in edetect_chain (the task makes them automatically)
and in studies of, for instance, the surface brightness of an extended source.
We will demonstrate how to make one using MOS1 data below. The procedure is
the same for MOS2 and PN.
Prior to making an exposure map, we will need to make an image in the energy band we are interested in. We also need the attitude file. For this example, we will make a soft link to the attitude file we made for edetect_chain and specify the 1-2 keV band.
| cd ..
mkdir expmap cd expmap ln -s ../ltcrv/m1_filt_time.fits ln -s ../source_detection/attitude.fits
evselect table=m1_filt_time.fits withimageset=yes imageset=m1_1-2keV.fits \
eexpmap imageset=m1_1-2keV.fits attitudeset=attitude.fits eventset=m1_filt_time.fits \
|
where parameters are are defined before, and
| expimageset - output exposure map
|
To do this with the GUI,
| 1) | Call evselect. |
| 2) | In the “General” tab, in table, enter the event file, m1_filt_time.fits. In the expression box, enter the filtering expression, PI in [1000:2000]. In the “Image” tab, check the withimageset box and in the imageset box, enter the output image name, m1_1-2keV.fits. Set xcolumn to X and ycolumn to Y. Set Binning to binSize, and both ximagebinsize and yimagebinsize to 40. |
| 3) | Click “Run”. |
| 4) | Call eexpmap. |
| 5) | In the imageset box, enter the image, m1_1-2keV.fits. In the eventset box, enter the event file, m1_filt_time.fits. In the attitudeset box, enter the attitude file, attitude.fits. In the pimin and pimax boxes, enter the minimum and maximum energies, 1000 and 2000, respectively. In the expimageset box, enter the name of the output file, m1_expmap_1-2keV.fits. |
| 6) | Click “Run”. |
Throughout the following, keep in mind that some parameters are instrument-dependent.
Specifically, the parameter specchannelmax should be set to 11999 for the MOS, and
20479 for the PN. Also, for the PN, the most stringent filter, FLAG==0, must be
included in the expression to get a high-quality, science-worthy spectrum. This
is not needed for the MOS.
There are some instances where tightening the selection requirements might be needed.
For example, if obtaining the best-possible spectral resolution is critical to your work, and the
corresponding loss of counts is not important, only the single pixel events should be
selected (PATTERN==0) for MOS and PN. If your observation is of a bright source
and pile up is a concern, you again might want to select only the single pixel events to
mitigate it (see §7.8 and §7.9 for
more on this topic).
In any case, we will again make a new directory for our files and display the filtered event file with ds9:
| cd ..
mkdir epicspec cd epicspec ln -s ../ltcrv/m1_filt_time.fits ln -s ../ltcrv/m2_filt_time.fits ln -s ../ltcrv/pn_filt_time.fits ds9 m1_filt_time.fits & |
Now we need to select the object whose spectrum we want to examine.
This can be done in ds9 by selecting Edit Region in the menu
bar and clicking on a source. Doing this will produce a circle
(extraction region), centered on the object. The circle's radius can be changed by
clicking on it and dragging it to the desired size. Adjust the size and position of
the circle until you are satisfied with the extraction region; then, double-click on
the region to bring up a window showing the center coordinates and radius of the
circle. For this example, we will choose the source at (X, Y) = (26188.5, 22816.5)
in physical units and set the extraction radius to 400 (again, in physical units).
While it is technically possible to use any positive number for specchannelmax,
it is recommended that this parameter should only be set to either 5 eV or 15 eV. The
reason for this derives from the event measurement thresholds on PN and MOS (5 eV and 3.26 eV,
respectively). Binning smaller than these thresholds will produce randomization, while
binning to 5 eV is a compromise between measurement detection and randomization. Binning
to 15 eV will place more emphasis on real measurement with only slight loss of resolution.
Regardless of which binning is used, it is recommended to use the same value for MOS and PN
if the spectra will be merged.
To extract the source spectra, type
| evselect table=m1_filt_time.fits energycolumn=PI filteredset=m1_filtered.fits \
expression='((X,Y) in CIRCLE(26188.5,22816.5,400))' spectrumset=m1_pi.fits \
spectralbinsize=5 withspecranges=yes specchannelmin=0 specchannelmax=11999
|
| evselect table=m2_filt_time.fits energycolumn=PI filteredset=m2_filtered.fits \
expression='((X,Y) in CIRCLE(26188.5,22816.5,400))' spectrumset=m2_pi.fits \
spectralbinsize=5 withspecranges=yes specchannelmin=0 specchannelmax=11999
|
| evselect table=pn_filt_time.fits energycolumn=PI filteredset=pn_filtered.fits \
expression='(FLAG==0) && ((X,Y) in CIRCLE(26188.5,22816.5,400))' \
spectrumset=pn_pi.fits spectralbinsize=5 withspecranges=yes specchannelmin=0 \
specchannelmax=20479
|
where parameters are as defined above, and
| energycolumn - energy column
spectrumset - name of output spectrum spectralbinsize - size of bin, in eV withspecranges - covering a certain spectral range specchannelmin - minimum of spectral range specchannelmax - maximum of spectral range |
To do this with the GUI for MOS1,
| 1) | Call evselect. |
| 2) | In the “General” tab, in table, enter the event file, m1_filt_time.fits.
Check keepfilteroutput and withfilteredset boxes.
In the filteredset box, enter the name of the event file output, m1_filtered.fits.
In the expression box, enter the filtering expression,
((X,Y) in CIRCLE(26188.5,22816.5,400)). In the “Spectrum” tab, confirm that withspectrumset is checked. In the spectrumset box, enter the output spectrum name, m1_pi.fits Set spectralbinsize to 5 and withspecranges to yes. Set specchannelmin and specchannelmax to 0 and 11999, respectively. |
| 3) | Click “Run”. |
The general procedure is the same for MOS2 and PN, with the exception that the specchannelmax
parameter for PN should be 20479.
For extracting a MOS background spectrum, it is recommended to select a region that is
on the same CCD, away from any sources. For extracting a PN background spectrum, it is
recommended to select a region that doesn't include columns that pass through the source
to avoid out-of-time events (this is particularly important for bright sources, and less
so for faint ones). The region should also have the same distance to the readout node as
the source region, i.e., they have the same RAWY values. For each CCD, RAWY's origin is
at the chip edge that corresponds to the outside of the array. So, the source and background
regions should have about the same distance to the outer edge of whichever CCD they are on.
If that isn't possible, choose a region that is on the same CCD as the source; and if that
isn't possible, choose a region in the same quadrant.
In our example dataset, there is plenty of suitable space for a background region on the
same CCD and with the same RAWY values. For a background spectrum from the MOS detectors,
we will choose an annulus around the source.
To extract the background spectra, type
evselect table=m1_filt_time.fits energycolumn=PI filteredset=m1_filtered_bkg.fits \
 expression= \
'((X,Y) in CIRCLE(26188.5,22816.5,1500)) &&! ((X,Y) in CIRCLE(26188.5,22816.5,500))' \
spectrumset=m1_pi_bkg.fits spectralbinsize=5 withspecranges=yes \
specchannelmin=0 specchannelmax=11999
evselect table=m2_filt_time.fits energycolumn=PI filteredset=m2_filtered_bkg.fits \
evselect table=pn_filt_time.fits energycolumn=PI filteredset=pn_filtered_bkg.fits \
|
where the keywords are as described above. To do this with the GUI, the same general
procedure that was used to extract the source spectrum can be used, with the parameter
values changed as needed, of course. The extracted regions for each detector are
shown in Figure 7.4. Note that in the PN image, the
source and background regions are on the same CCD, despite being separated by a
what appears to be a chip gap - it is not, in fact a chip gap, but rather a
noisy column that was removed.
|
Depending on how bright the source is and what modes the EPIC detectors are in,
event pile up may be a problem. Pile up is analogous to saturation in optical/IR
detectors, and it occurs when a source is so bright that incoming X-rays strike
two neighboring pixels or the same pixel in the CCD more than once per readout cycle.
X-ray loading is a related phenomenon, occurring when the X-ray count rate is
sufficiently high that too many events per pixel are detected during the offset
map calculation.
In the case of pile up, the energies of multiple events are added together to form
one event. If this happens sufficiently often, 1) the spectrum
will appear to be harder than it actually is (i.e., it will be artificially shifted to
higher energies), 2) the count rate will be underestimated, since multiple events
will be undercounted, and 3) the measured event pattern distribution will not agree
with what is expected from modeling - more specifically, there will be too many
multiple-pixel patterns and too few single-pixel ones. Given the impact that it can
have on the data, it is important to remove it or mitigate its effects when possible.
We recommend to always check for it.
If a source has a count rate 
0.5 c/s for the MOS in full frame mode
or 2 c/s for the PN in full frame mode (see the XMM-Newton Users
Handbook for information on other modes), pile up and X-ray loading will affect the
data quality. For extremely bright sources, it will be immediately obvious upon inspection
of the EPIC data - there will appear to be “hole” in the center of the source
in the MOS cameras. This is from the observatory's onboard event reconstruction software,
which removes complicated patterns involving many pixels. (For PN, event reconstruction
happens during on-ground data processing with SAS.)
But very often, pile up is not obvious. Luckily, SAS has task to help us look
for it, epatplot.
Please note that there is a bug in this version which may prevent the task from
finding a font it needs when making the plot, resulting in small rectangles being
printed instead of the correct characters. This will be fixed in future versions.
For now, there is an easy way around it: when calling epatplot from the
command line, set the parameter withsrcxy to no.
Note that this task requires as input the event files created when the spectrum was made, not the usual time-filtered event file. To check for pile up in our example spectrum in MOS1, type
| epatplot set=m1_filtered.fits plotfile=m1_epat.pdf useplotfile=yes \
withbackgroundset=yes backgroundset=m1_filtered_bkg.fits withsrcxy=no
|
where
| set - input events file
plotfile - output pdf file useplotfile - send the plot to a file? withbackgroundset - use background event set for background subtraction? backgroundset - name of background event file withsrcxy - also plot real-valued SRCPOSX and SRCPOSY positions? (bug fix) |
To do this with the GUI,
| 1) | Call epatplot. |
| 2) | In the “0” tab, in the set box, enter the name of the event file that was made when the spectrum was extracted, m1_filtered.fits. We want to send the output to a pdf file, so set useplotfile to yes, and enter the file name in the plotfile box. We will use m1_epat.pdf. |
| 3) | In the “1” tab, set withbackgroundset to yes and enter the name of the event file that was made when the background spectrum was made, m1_filtered_bkg.fits. Toggle withsrcxy to no. |
| 4) | Click “Run”. |
The output of epatplot is a pdf file, m1_epat.pdf which may be
viewed with standard pdf viewers. It contains two graphs describing the distribution
of counts as a function of PI channel; see Figure 7.5
(left).
A few words about interpretting the plots are in order. In
Figure 7.5 (left), the top plot is the distribution
of counts versus PI channel for each pattern class (single, double, triple, quadruple; see
§4.1 for details on pattern types) in the spectrum,
and the bottom is the expected pattern distribution (smooth lines) plotted over
the observed distribution (histogram). The lower plot shows the model
distributions for single and double events and the observed distributions. It also
gives the ratio of observed-to-modeled events with 1-
uncertainties for single and
double pattern events over a given energy range. (The default is 0.5-2.0 keV; this can be
changed with the pileupnumberenergyrange parameter.) If the data are not piled up,
there will be good agreement between the modeled and observed single and double event
pattern distributions in this lower plot. Also, the observed-to-modeled fractions for
both singles and doubles will be unity, within errors. In contrast, if there is pile up
(or X-ray loading), there will be a clear difference between the modeled and observed pattern
distributions, and the observed-to-modeled fraction for singles will be less than 1.0, and for doubles,
it will be greater than 1.0. If the data are too sparse
for the task to produce meaningful output, the lower plot will show only the model
distribution, without the observed data histogram.
Finally, when examining the plots, it should noted that the observed-to-modeled fractions
can be inaccurate. Therefore, the agreement between the modeled and observed single and
double event pattern distributions should be the main factor in determining if an observation
is affected by pile up or not.
In Figure 7.5 (left), we can see that
our Lockman Hole source is indeed very faint - too faint for epatplot to
give us reliable statistics. The observed-to-modeled fractions for both singles and
doubles agree with unity. We need not worry about the effects of pile up here!
For contrast, in Figure 7.5 (right), we
show the output for a much brighter source from a different observation.
The lower plot shows a clear discrepancy between the modeled and
observed pattern distributions, especially at 2 keV, with there being too
few singles patterns than expected and too many doubles. There clearly is
pile up at these high energies.
There are two ways to deal with pile up in an EPIC camera observation.
First, the innermost regions of a source can be excised (as they are the most heavily
piled up). If doing this, it is recommended to take an iterative approach: remove the source
core, make a spectrum, check to see if there is still pile up, and repeat, each
time removing a slightly larger region, until the model and observed distribution
functions agree. This technique can be used on both Imaging mode and Timing mode data,
the only difference being the shape of the excised region, as Imaging mode for
a point source will have a circular region file and Timing mode data will have
a box-shape region file (this is because in Timing mode, there is spatial
information only on the RAWX axis, since it is lost on the RAWY axis due to
continuous shifting and collapsing of rows as they are quickly read out (an image
of Timing mode data is in Figure 8.2). It is good to be
aware that removing too small
a region with respect to the instrumental pixel size (1.1” for the MOS, 4.1” for
the PN) can introduce systematic inaccuracies when calculating the source flux;
these are less than 4%, and decrease to less than 1% when the excised region is
more than 5 times the instrumental pixel half-size. In any case, be certain that the
excised region is larger than the instrumental pixel size!
Second, the event file can be filtered so that only single pixel events (pattern 0)
are used, as these are less sensitive to pile up than other patterns.
Let's return now to the topic of X-ray loading. This has the same general origin
as pile up (i.e., observing a source with a high count rate), but it occurs when
offset maps are being made on board the spacecraft at the start of an
exposure. The offset per pixel is calculated from a sample of frames
from which the three highest and three lowest energy values per pixel
are excluded. If the count rate is high, this method cannot exclude all
the X-ray photons that should be excluded, and thus produces an offset
for a given pixel, in addition to its dark current. The severity of X-ray
loading depends on the energies of the X-ray events left in the sample after
the three highest and lowest values are excluded. Because the offset per pixel
is too high, the spectrum is artificially shifted to lower energies, and the
there will be too many single-pixel patterns and too few multiple-pixel
ones - in other words, the opposite of pile up. The MOS detectors
have a lower quantum efficiency than the PN, so this isn't usually a problem
in MOS images, but it can be for PN Imaging mode data. (It is automatically
corrected in PN Timing and Burst modes.) A pattern plot from a PN observation
that shows X-ray loading at 
6 keV and pile up at 6 keV is in
Figure 7.6. If you're working with an observation
where this is a problem, it is straightforward to make X-ray loading-corrected
event files with either epproc or epchain:
| epproc runepxrlcorr=yes |
or
| epchain runepxrlcorr=yes |
and then apply the standard filters, temporal filters (if needed), and re-extract the spectrum as usual.
|
|
Now that we are confident that our spectrum is not piled up, we can continue by
finding the source and background region areas. This is done with the task backscale,
which takes into account any bad pixels or chip gaps, and writes the result into the BACKSCAL
keyword of the spectrum table. Alternatively, we can skip running backscale, and use
a keyword in arfgen. We will show both options for the curious.
To find the source and background extraction areas explicitly:
| backscale spectrumset=m1_pi.fits badpixlocation=m1_filt_time.fits
backscale spectrumset=m1_pi_bkg.fits badpixlocation=m1_filt_time.fits backscale spectrumset=m2_pi.fits badpixlocation=m2_filt_time.fits backscale spectrumset=m2_pi_bkg.fits badpixlocation=m2_filt_time.fits backscale spectrumset=pn_pi.fits badpixlocation=pn_filt_time.fits backscale spectrumset=pn_pi_bkg.fits badpixlocation=pn_filt_time.fits |
where
| spectrumset - name of the spectrum file
badpixlocation - the file containing the bad pixel locations (the event file) |
To do this with the GUI, for the source in MOS1,
| 1) | Call backscale. |
| 2) | In the “Main” tab, in the spectrumset box, enter the spectrum, m1_pi.fits. |
| 3) | In the “Effects” tab, in the badpixlocation box, enter the event file, m1_filt_time.fits. |
| 4) | Click “Run”. |
Repeat these steps with the appropriate changes for the background spectrum, then do the same for MOS2 and PN.
Next, we need to reformat the detector response by making redistribution matrix files (RMFs) and ancillary response files (ARFs). To make the RMFs, type:
| rmfgen rmfset=m1_rmf.fits spectrumset=m1_pi.fits
rmfgen rmfset=m2_rmf.fits spectrumset=m2_pi.fits rmfgen rmfset=pn_rmf.fits spectrumset=pn_pi.fits |
where
| rmfset - output file
spectrumset - spectrum file |
To do this with the GUI, for the source in MOS1,
| 1) | Call rmfgen. |
| 2) | In the “Main” tab, in the spectrumset box, enter the spectrum file name, m1_pi.fits. In the rmfset box, enter the output RMF file name, m1_rmf.fits. |
| 3) | Click “Run”. |
Repeat these steps with the appropriate changes for MOS2 and PN.
We will use the RMFs, spectra, and event files to make the ARFs:
| arfgen arfset=m1_arf.fits spectrumset=m1_pi.fits withrmfset=yes \
rmfset=m1_rmf.fits withbadpixcorr=yes badpixlocation=m1_filt_time.fits
|
| arfgen arfset=m2_arf.fits spectrumset=m2_pi.fits withrmfset=yes \
rmfset=m2_rmf.fits withbadpixcorr=yes badpixlocation=m2_filt_time.fits
|
| arfgen arfset=pn_arf.fits spectrumset=pn_pi.fits withrmfset=yes \
rmfset=pn_rmf.fits withbadpixcorr=yes badpixlocation=pn_filt_time.fits
|
where parameters are as defined above, and
| arfset - output ARF file name
spectrumset - input spectrum file name withrmfset - flag to use the RMF to specify the energy grid rmfset - RMF file created by rmfgen withbadpixcorr - flag to include the bad pixel correction badpixlocation - file containing the bad pixel information; should be set to the event file from which the spectrum was extracted.
|
To do this with the GUI, for the source in MOS1,
| 1) | Call arfgen. |
| 2) | In the “Main” tab, set the arfset parameter to the ARF file name, m1_arf.fits. Set the spectrumset parameter to the spectrum file, m1_pi.fits. |
| 3) | In the “Effects” tab, set the badpixlocation keyword to the event file name from which the spectrum was extracted, m1_filt_time.fits. |
| 4) | In the “Calibration” tab, check the withrmfset box and set the rmfset keyword to the RMF file name, in this case, m1_rmf.fits. |
| 5) | Click “Run”. |
Repeat these steps with the appropriate changes for MOS2 and PN.
If we had not run the backscale task, we could have used an option in arfgen
to run it, by simply adding setbackscale=yes to the arfgen call on the
command line, or by going to the “Backscale” tab and toggling the keyword to yes in the GUI.
At this point, the spectrum is ready to be analyzed. Information on how to prepare the spectrum for fitting can be found in §12.1, including how to group it using ftools. However, SAS can also be used to group spectra, as shown here. For our first example, we will rebin the MOS1 spectrum to have a minimum of 30 counts in each background-subtracted channel. We will also assign the response files to the appropriate header keywords.
| specgroup spectrumset=m1_pi.fits mincounts=30 addfilenames=yes \
rmfset=m1_rmf.fits arfset=m1_arf.fits backgndset=m1_bkg_pi.fits groupedset=m1_pi_bin30.fits
|
This task can also rebin by signal-to-noise ratio. Here, we will rebin so that each channel will have a minimum S/N = 5:
| specgroup spectrumset=m1_pi.fits minSN=5 groupedset=m1_pi_sn5.fits |
To aid in scripting, the data preparation commands from §6.1 and §6.2 and the commands used in this chapter are consolidated below.
| cd ODF
setenv SAS_ODF `pwd` ; setenv SAS_ODFPATH `pwd` cifbuild setenv SAS_CCF `pwd`/ccf.cif odfingest setenv SAS_ODF `pwd`/*SUM.SAS
cd ..
mkdir ltcrv
cd ..
cd ..
cd ..
|
![\includegraphics[scale=0.15]{LockmanHole_0123700101_ltcrv_fv_3.eps}](img19.png)
![\includegraphics[scale=0.15]{full-filtered-3.eps}](img23.png)
![\includegraphics[scale=0.15]{LockmanHole_0123700101_mosaicked_image.eps}](img25.png)
![\includegraphics[scale=0.15]{epic-spectrum-regions.eps}](img27.png)
![\includegraphics[scale=0.5]{compare_epic_pileup.eps}](img28.png)
![\includegraphics[scale=0.35]{4u1608-522_epat.eps}](img29.png)