skip to content
 
XMM-Newton Guest Observer Facility

XMM-NEWTON DATA ANALYSIS WITH PERL SCRIPTS IN HERA


The GOF has developed Perl scripts in an effort to simplify the EPIC data reduction process and make XMM data more accessible to astrophysicists and students who may not be very familiar with the details of working with SAS. These have been implemented in Hera. A short guide to their use for both the GUI and the Command Window is available below, using a Lockman Hole observation (ObsID 0147511601) for the examples. All the steps given here are the same as in the ABC Guide.

Scripts for RGS and OM are currently in development.

EPIC Scripts
A Walk-Through with the GUI
A Walk-Through with the Command Window


EPIC Scripts


Short descriptions for each script are given below, as are the input parameters, both mandatory and optional . Possible values for each input parameter are in parentheses.
  • sas_reprocess_epic: repipeline the data and apply standard filters. process_mos (Y/N), process_pn (Y/N)
  • sas_make_expmap_epic: make exposure maps based on default energy ranges or user-supplied ranges. event_file (text file), energy_var (Y/N), energy_file (text file)
  • sas_make_bkg_spectrum_epic: make a background spectrum. event_file (text file), position_file (text file)
  • sas_make_filteredevents_epic: remove flares from event files. event_files (text file), times_file (text file)
  • sas_make_image_epic: make images for display in ds9 or fv. event_files (text file), position_var (Y/N), position_file (text file)
  • sas_make_lc_epic: make light curves. event_file (text file), lcbin (float or integer), position_var (Y/N), position_file (text file)
  • sas_make_spectrum_epic: extract spectra, make RMFs and ARFs, check for pile-up. event_file (text file), position_file (text file)

A Walk-Through with the GUI

The scripts are located in the "Experimental" folder, under "XMM-SAS" in the "Available Tools" window of the interface. As always when working with Hera, be sure to call tasks from your working directory.

  1. Download the data to your Hera account and unzip the files in the ODF directory.
  2. Prepare the data, then reprocess and filter them. Highlight the ODF directory, then call "sas_reprocess_epic". In the pop-up window, verify the working directory is correct, and select which cameras to process. The output event files will appear in the directory "proc" and follow the naming convention "instrument_evt2.fits". sas_reprocess_epic will automatically update your environment variables to point to the *SUM.SAS files it makes. (You may need to refresh the directory listing to see the new files and directory.)

  3. Download the event files to your local machine for easy examination and prepare some text files for future use. (Files are copied between the host and client by right-clicking and dragging.) We're interested in data from all three detectors, so make a file listing all their event files on your local machine, then upload it to your Hera account:

    MyComputer> ls *evt2.fits > events.txt

    There are two somewhat bright sources in the field. We can list these in a text file, using the convention:

    unit
    x y inner_radius outer_radius
    where
    unit = unit the coordinates and radii are in; can be either d (decimal degrees) or p (pixels)
    inner_radius = inner radius of annulus region to extract. Set this to 0 to extract a circular region.
    outer_radius = outer radius of annulus region to extract. This is equal to the radius of a circular region.
    So the region file in degree space would look like:

    > more coords.txt
    d
    163.16561 57.409134 0 0.005
    163.32071 57.597477 0 0.005

    If you wanted to use pixel units instead, the file would look like this:

    > more coords.txt
    p
    25748.549 18870.529 0 600
    19765.33 32438.468 0 600

    We can also make a file denoting background regions for these sources:

    > more coords_bkg.txt
    d
    163.16561 57.409134 0.005 0.015
    163.32071 57.597477 0.005 0.015

    If you wanted to use pixel units instead, the file would look like this:

    > more coords_bkg.txt
    p
    25748.549 18870.529 360 1080
    19765.33 32438.468 360 1080

    When the region files have been made, upload them to the "proc" directory in Hera.

  4. Make images with sas_make_image_epic. In the pop-up window, verify the working directory is correct, and enter the file that contains the event file names, "events.txt". The full path is not needed. By default, this task will make an image of the full field of view, but it can also make images of specific regions. This can be done by selecting this option in the pop-up and entering the region file name. The output will be in the "image" directory and will follow the naming convention "instrument_img.fits". If you use a region file, the images of each source will be numbered in the order in which they are given in the file.

  5. Make light curves to see if there are any flares that need to be removed with sas_make_lc_epic. In the pop-up window, verify the working directory is correct, and enter the file that contains the event file names, "events.txt". The full path is not needed. Enter the time bin size; for this dataset, 10 or 100 is a good choice. By default, this task will make a light curve of the full field of view, but it can also make light curves of specific regions by selecting this option in the pop-up and supplying the region file name. If there is more than one region, the light curves of each object will be numbered in the order in which they are given in the file. The output will be in the "lightcurve" directory.

  6. Examine the lightcurves. If necessary, filter the event files to remove flares. This is done by making a text file on your local machine that lists the starting and ending BAD times, one per line, and uploading this to your Hera account. For our example case, there are several time ranges contaminated with flares:

    > more bad_times.txt
    1.54854e8 1.54865e8
    1.54870e8 1.54873e8
    1.54912e8 1.54921e8
    1.54926e8 1.54952e8

    To remove these time ranges, make sure you are in the "proc" directory and call sas_make_filteredevents_epic. In the pop-up window, verify that the working directory is correct, then enter the event file list and the bad times list. There will be several "DssBlockClash" warnings; these can be ignored. The output will be placed in the working directory and follow the naming convention "instrument_flt_evt2.fits". Remember to update the events file list on your local machine with the filtered files and upload the new file list!

  7. Use sas_make_spectrum_epic to extract a spectrum, test it for pile-up, and produce response and ancillary files. In the pop-up window, verify that the working directory is correct, then enter the event file list and the region file. The output will be placed in the directory "spectrum". Each source in the region file will have a spectrum (*.pi), a postscript file comparing the measured and modeled count rates produced with the SAS task epatplot, and response files (*.rmf and *.arf). These products will be numbered in the order that the sources are given in the file.

  8. Make a background spectrum with sas_make_bkg_spectrum_epic. In the pop-up window, verify that the working directory is correct, then enter the event file list and the background region file, like what was shown in step 3. The output spectrum will be placed in the "spectrum" directory. Each spectrum will be numbered in the order that the sources are given in the region file. Unlike sas_make_spectrum_epic, this task only extracts the spectrum, and does not check for pile up or generate response files.

  9. Make exposure maps with sas_make_expmap_epic. The default energy range for exposure maps is 2-4.5 keV. However, if you are interested in a different energy range (or several ranges), this is easily accomodated: make a text file on your local machine that contains the desired minimum and maximum energies (in eV), one per line, and upload it to your Hera account's "proc" directory. For instance, the file "energy.txt" may look like this:

    MyComputer> more energy.txt
    1000 2000
    2000 3000

    After uploading the text file, call sas_make_expmap_epic. In the pop-up window, verify that the working directory is correct, then enter the event file list and the energy range list. The output is placed in the "expmap" directory and follows the naming convention "instrument_expmap_LowEnergy-HighEnergykeV.fits". So for our example, six exposure maps will be made: m1_expmap_1-2keV.fits, m1_expmap_2-3keV.fits, m2_expmap_1-2keV.fits, m2_expmap_2-3keV.fits, pn_expmap_1-2keV.fits, and pn_expmap_2-3keV.fits.

With these data products, further analysis can be done with XSpec or CIAO, as shown in the
ABC Guide.

A Walk-Through with the Command Window

As always when working with Hera, be sure to call up the Command Window from your working directory. If you want to switch directories, close the Window you are working in, highlight the desired directory in the GUI, and open a new Window.

  1. Download the data to your Hera account and unzip the files in the ODF directory.
  2. Prepare the data, then reprocess and filter them. Highlight the ODF directory and bring up a Command Window. For this example, we want to reprocess data from the PN and MOS detectors, so enter

    > sas_reprocess_epic process_mos=y process_pn=y

    The filtered event files will appear in the directory "proc" and follow the naming convention "instrument_evt2.fits". Hera will automatically update your environment variables to point to the new *SUM.SAS file.

  3. Download the event files to your local machine for easy examination and prepare some text files for future use. (Files are copied between the host and client by right-clicking and dragging.) We're interested in data from all three detectors, so make a file listing all their event files on your local machine, then upload it to your Hera account:

    MyComputer> ls *evt2.fits > events.txt

    There are two somewhat bright sources in the field. We can list these in a text file, using the convention:

    unit
    x y inner_radius outer_radius
    where
    unit = unit the coordinates and radii are in; can be either d (decimal degrees) or p (pixels)
    inner_radius = inner radius of annulus region to extract. Set this to 0 to extract a circular region.
    outer_radius = outer radius of annulus region to extract. This is equal to the radius of a circular region.
    So the region file in degree space would look like:

    > more coords.txt
    d
    163.16561 57.409134 0 0.005
    163.32071 57.597477 0 0.005

    If you wanted to use pixel units instead, the file would look like this:

    > more coords.txt
    p
    25748.549 18870.529 0 600
    19765.33 32438.468 0 600

    We can also make a file denoting background regions for these sources:

    > more coords_bkg.txt
    d
    163.16561 57.409134 0.005 0.015
    163.32071 57.597477 0.005 0.015

    If you wanted to use pixel units instead, the file would look like this:

    > more coords_bkg.txt
    p
    25748.549 18870.529 360 1080
    19765.33 32438.468 360 1080

    When the region files have been made, upload them to the "proc" directory in Hera.

  4. Make images with sas_make_image_epic. Close the current Command Window, highlight the "proc" directory in the GUI, then open a new Command Window and enter

    > sas_make_image_epic event_files=events.txt position_var=n

    To make images of the regions in the coordinate files, enter

    > sas_make_image_epic event_files=events.txt position_var=y position_file=coords.txt

    The output will be in the "image" directory and will follow the naming convention "instrument_img.fits". If you use a region file, the images of each source will be numbered in the order in which they are given in the file.

  5. Make light curves to see if there are any flares that need to be removed with sas_make_lc_epic. For this dataset, a time bin size of 10 or 100 is a good choice. For this example, we will let the time bin size be 10. By default, this task will make a light curve of the full field of view, but it can also make light curves of specific regions by setting "position_var" to "y" and supplying the region file name. If there is more than one region, the light curves of each object will be numbered in the order in which they are given in the file. The output will be in the "lightcurve" directory.

    > sas_make_lc_epic event_file=events.txt lcbin=10 position_var=y position_file=coords.txt

  6. Examine the lightcurves. If necessary, filter the event files to remove flares. This is done by making a text file on your local machine that lists the starting and ending BAD times, one per line, and uploading this to your Hera account. For our example case, there are several time ranges contaminated with flares:

    > more bad_times.txt
    1.54854e8 1.54865e8
    1.54870e8 1.54873e8
    1.54912e8 1.54921e8
    1.54926e8 1.54952e8

    To remove these time ranges, enter:

    > sas_make_filteredevents_epic events_files=events.txt time_file=bad_times.txt

    There will be several "DssBlockClash" warnings; these can be ignored. The output will be placed in the working directory and follow the naming convention "instrument_flt_evt2.fits". Remember to update the events file list on your local machine with the filtered files and upload the new file list!

  7. Use sas_make_spectrum_epic to extract a spectrum, test it for pile-up, and produce response and ancillary files by entering:

    > sas_make_spectrum_epic event_file=events.txt position_file=coords.txt

    The output will be placed in the directory "spectrum". Each source in the region file will have a spectrum (*.pi), a postscript file comparing the measured and modeled count rates produced with the SAS task epatplot, and response and ancillary files (*.rmf and *.arf). These products will be numbered in the order that the sources are given in the file.

  8. Make a background spectrum with sas_make_bkg_spectrum_epic by entering:

    > sas_make_bkg_spectrum_epic event_file=events.txt position_file=coords_bkg.txt

    The output spectrum will be placed in the "spectrum" directory. Each spectrum will be numbered in the order that the sources are given in the region file. Unlike sas_make_spectrum_epic, this task only extracts the spectrum, and does not check for pile up or generate response files.

  9. Make exposure maps with sas_make_expmap_epic. If there are particular energy ranges you are interested in, you can enter a text file with those ranges. The file must list the minimum and maximum energies (in eV), one per line. Make it on your local machine and upload it to the "proc" directory in your Hera account. For instance,

    MyComputer> more energy.txt
    1000 2000
    2000 3000

    will produce two exposure maps for each event file listed in events.txt, over the ranges 1-2 keV and 2-3 keV. If no energy file is entered, the default range 2-4.5 keV is used. The output is placed in the "expmap" directory and follows the naming convention "instrument_expmap_LowEnergy-HighEnergykeV.fits". So for our example, six exposure maps will be made: m1_expmap_1-2keV.fits, m1_expmap_2-3keV.fits, m2_expmap_1-2keV.fits, m2_expmap_2-3keV.fits, pn_expmap_1-2keV.fits, and pn_expmap_2-3keV.fits.

    > sas_make_expmap_epic event_file=events.txt energy_var=y energy_file=energy.txt

With these data products, further analysis can be done with XSpec or CIAO, as shown in the
ABC Guide.
If you have any questions concerning XMM-Newton send e-mail to xmmhelp@lists.nasa.gov