NAME

aepipeline - Suzaku XIS/HXD reprocessing tool

SYNOPSIS

aepipeline indir=<Input Directory> outdir=<Output Directory> steminputs=<Stem for input files> entry_stage=<Entry stage> exit_stage=<Exit stage> instrument=<Instrument>

DESCRIPTION

This task duplicates most of the Suzaku pipeline (not trend data). It allows the user to run all or part of the pipeline processing and to vary the calibration files and filtering (screening) criteria used. A number of other pipeline processing parameters can also be changed.

Pipeline Stages

The pipeline is divided into 2 stages:

  1. Calibration
  2. Data screening
Each stage may be run singly or in combination with the preceeding stages. This is controlled by the entry_stage and exit_stage parameters.

HXD Stage 1 consists of the following steps:

  1. Generate a telemetry un-saturated GTI (hxdgtigen)
  2. Run hxdtime
  3. Run hxdpi
  4. Run hxdgrade

XIS Stage 1 consists of the following steps:

  1. Generate a telemetry un-saturated GTI (xisgtigen)
  2. Run xistime
  3. Run xiscoord
  4. Run xisputpixelquality
  5. Run xispi

The data screening (Stage 2) is identical to that in the production Suzaku pipeline, when default parameters are used. For details on the default screening applied to the XIS and HXD events (respectively), see:

To change the screening criteria from the default, use the xis[0-3]_expr, hxd_{pse,gso,pin}_expr, xis[0-3]_grade and xis[0-3]_psum_grade parameters.

Instruments

aepipeline allows the processing of data on an instrument by instrument basis. This is controlled by the instrument parameter. Possible values are:

More than one value may be specified at one time.
Combination of GSO and PIN is allowed, is equivalent to HXD.
Combinations of two or more of XIS0, XIS1, XIS2, XIS3 are allowed.
Combination of XIS0, XIS1, XIS2, XIS3 is equivalent to XIS.

INPUT

The input to aepipeline is specified using (at minimum) the indir parameter. This should be specified as the top level sequence directory, e.g.:

aepipeline indir=/path/to/100039010 ...

Paths to specific housekeeping and satellite data files can be specified using the attitude, housekeeping, extended_housekeeping, makefilter, orbit and timfile parameters. But this should NOT be necessary if indir is specified correctly.

NOTE: Errors can occur if a subdirectory of a sequence directory is used for the indir parameter or if the output is stored within the sequence directory.

OUTPUT

Report

An output report is produced with the name "<steminputs>_aepipeline_report.txt". This lists all processing done in any particular run of aepipeline. The production of this report is controlled by the report parameter. (Default = "yes".) The level of detail of this report is controled by the chatter parameter. (Default = "2", range 0 - 5.)

NOTE: Setting chatter to "3" or above can produce very large output reports.

Filenames, etc.

The number of output files depends on both pipeline processing stage(s) and the instrument(s) selected. All output files are written to the directory specified by the outdir parameter. The archive directory structure is NOT reproduced (i.e. all output files will be in a single directory).

The names of files produced are the same as those found in the HEASARC archive. However the usual "aeXXXXXXXXX" prefix, where "XXXXXXXXX" is the sequence number, can be replaced by a character string set by the stemoutputs parameter. This defaults to the value set by the steminputs parameter.

PARAMETERS

indir [string]
Directory containing the input data. This should be the full, or relative path to the top level sequence directory, e.g.:

    /path/to/805062010/

NOTE: Problems can arise if a sub-directory of the top level sequence directory is used.

outdir [string]
Output directory used for all output data, as well as the report file. If clobber is not set, and this directory already exists, the task will fail. This should NEVER be a sub-directory of the input directory (the top level sequence directory).

CAUTION: If clobber is set, and this directory already exists, then this task will overwrite files as needed in this directory.

steminputs [string]
Stem for FITS input files, e.g. ae100039010. This string is used as the base filename for finding input files. For example, if steminputs=ae100039010, then to find the attitude file, aepipeline matches the regular expression /ae100039010\.att(\..+)?$/ against all files found in the indir directory.

When doing this matching, aepipeline will prefer files that match the original archive directory structure over files with the same name that do not. For example, if the files:

    /path/to/805062010/hxd/event_uf/ae805062010hxd_0_wel_uf.evt

    /path/to/805062010/hxd/event_uf_2/ae805062010hxd_0_wel_uf.evt

both exist, aepipeline will use the first file, and not the second. If two files of the same name are found and neither one resides in an archive directory structure, aepipeline will exit with an error.

NOTE: The above file finding issues can be avoided if the downloaded archive data set is left "pristine", by specifying an output directory OUTSIDE of the input directory tree. This is strictly enforced as of V1.1.0 of aepipeline.

entry_stage = 1 [1|2]
Entry stage, 1 or 2.

Stage 1: Re-calibrate unfiltered event files.

Stage 2: Start from existing unfiltered event files.

exit_stage = 2 [1|2]
Exit stage, 1 or 2.

Stage 1: Produces calibrated unfiltered event files.

Stage 2: Produces screened event files.

instrument = ALL [string]
Comma delimited list of which instruments to process. List can include the following values:

More than one value may be specified at one time:

(stemoutputs = DEFAULT) [string]
Base (stem) output name used for creating output files. If set to "DEFAULT", then steminputs will be used.

(clobber = no) [bool]
Clobber existing output files?

(chatter = 2) [int]
Chattiness level.

0 - Essentially nothing will be produced.

5 - Debugging mode with large quantities of output.

NOTE: Setting chatter=5 can generate a huge report file (see report parameter) and a very large amount of terminal output. It is not recommended for general use.

(history = yes) [bool]
Add HISTORY keywords to output FITS files?

(report = yes) [bool]
Produce separate report file for run?

(remove_temp_files = yes) [bool]
Remove temporary files?

NOTE: If set to 'no' and entry_stage is set to 1, roughly 4 to 5 times the disk space will be required for output.

(verify_input = no) [bool]
Run ftverify on input files?

(seed = 7) [int]
Seed for random number generators.

(xisconflist = $ENV{LHEA_DATA}/aexisconf.list) [filename]
XIS discriminator threshold letter code lookup table. This list determines how screened XIS event files are named.

(leapfile = CALDB;$ENV{LHEA_DATA}/leapsec.fits) [filename]
Leap second calibration file.

(attitude = DEFAULT) [filename]
Attitude file. A value of "DEFAULT" will use the "<steminputs>.att" file found in the indir. A value of "EULER" will use fixed Euler angles for the attitude of the satellite (see ea1, ea2, ea3 parameters).

(housekeeping = DEFAULT) [filename]
Housekeeping file. A value of "DEFAULT" will use the "<steminputs>.hk" file found in the indir.

(extended_housekeeping = DEFAULT) [filename]
Extended housekeeping file. A value of "DEFAULT" will use the "<steminputs>.ehk" file found in the indir.

(makefilter = DEFAULT) [filename]
Makefilter file. A value of "DEFAULT" will use the "<steminputs>.mkf" file found in the indir.

(orbit = DEFAULT) [filename]
Orbit file. A value of "DEFAULT" will use the "<steminputs>.orb" file found in the indir.

(timfile = DEFAULT) [filename]
Packets Timing file. A value of "DEFAULT" will use the "<steminputs>.tim" file found in the indir.

(hxd_start = 77604208E+08) [real]
HXD CALDB start time, in seconds. You should not need to change this parameter.

(hxdpi_old = no) [bool]
Use hxdpi_old version instead of hxdpi?

(hxd_gsoght = CALDB) [filename]
HXD GSO gain history table - ONLY used for hxdpi_old.

(hxd_gsogpt = CALDB) [filename]
HXD GSO gain parameter table - ONLY used for hxdpi (current version).

(hxd_gsolin = CALDB) [filename]
HXD GSO linearity table.

(hxd_gsopsd = CALDB) [filename]
PSD selection file.

(hxd_pinghf = CALDB) [filename]
HXD PIN gain history file.

(hxd_pinlin = CALDB) [filename]
HXD PIN linearity table.

(hxd_pinthr = CALDB) [filename]
HXD PIN threshold file.

(WPU = "0123") [string]
WPU board IDs.

(xis_start = 77120000E+08) [real]
XIS CALDB start time, in seconds. You should not need to change this parameter.

(xis0_badcolum = CALDB) [filename]
XIS0 bad column file.

(xis0_calmask = CALDB) [filename]
XIS0 calmask file.

(xis0_makepi = CALDB) [filename]
XIS0 trail file.

(xis0_teldef = CALDB) [filename]
XIS0 telescope definition file.

(xis1_badcolum = CALDB) [filename]
XIS1 bad column file.

(xis1_calmask = CALDB) [filename]
XIS1 calmask file.

(xis1_makepi = CALDB) [filename]
XIS1 trail file.

(xis1_teldef = CALDB) [filename]
XIS1 telescope definition file.

(xis2_badcolum = CALDB) [filename]
XIS2 bad column file.

(xis2_calmask = CALDB) [filename]
XIS2 calmask file.

(xis2_makepi = CALDB) [filename]
XIS2 trail file.

(xis2_teldef = CALDB) [filename]
XIS2 telescope definition file.

(xis3_badcolum = CALDB) [filename]
XIS3 bad column file.

(xis3_calmask = CALDB) [filename]
XIS3 calmask file.

(xis3_makepi = CALDB) [filename]
XIS3 trail file.

(xis3_teldef = CALDB) [filename]
XIS3 telescope definition file.

(anl_profile = yes) [bool]
Enable ANL profiling.

(anl_verbose = -1) [int]
ANL verbosity setting. (-1 for full, 0 for minimal)

(event_freq = 100000) [int]
Event row counting frequency in Suzaku tools.

(num_event = -1) [int]
Number of events to process (-1:all, 0:exit).

(time_convert_mode = 4) [int]
HxdTime2aetime mode.: 1, 2, 3, 4.

1: use internal function in interpolation (TI to aetime).

2: use telemetry aetime, and simply add sub_aetime.

3: use telemetry aetime, and interpolate event aetime

4: use astetool, aste_ti2time() with tim FITS file.

(use_pwh_mode = no) [bool]
Use HXD_WEL_PWH column in HK FITS or not.

(bstgti = no) [bool]
Tool Specific Parameter - XISTIME.

XISTIME - Generate GTI for the XIS burst option without approximation?

(pointing = KEY) [KEY|USER]
Tool Specific Parameter - XISCOORD.

XISCOORD - Pointing type, KEY/USER.

(ea1 = 0.0) [real]
Tool Specific Parameter - XISCOORD.

XISCOORD - 1st XYZ-Euler angle (deg), used when attitude=EULER.

(ea2 = 0.0) [real]
Tool Specific Parameter - XISCOORD.

XISCOORD - 2nd XYZ-Euler angle (deg), used when attitude=EULER.

(ea3 = 0.0) [real]
Tool Specific Parameter - XISCOORD.

XISCOORD - 3rd XYZ-Euler angle (deg), used when attitude=EULER.

(ref_alpha = 0.0) [real]
Tool Specific Parameter - XISCOORD.

XISCOORD - R.A. of the sky reference position, used when pointing=USER.

(ref_delta = 0.0) [real]
Tool Specific Parameter - XISCOORD.

XISCOORD - DEC. of the sky reference position, used when pointing=USER.

(ref_roll = 0.0) [real]
Tool Specific Parameter - XISCOORD.

XISCOORD - Roll angle of the sky reference, used when pointing=USER.

(aberration = no) [bool]
Tool Specific Parameter - XISCOORD.

XISCOORD - Correct aberration.

(enable_scipixq = yes) [bool]
Tool Specific Parameter - XISPUTPIXELQUALITY.

XISPUTPIXELQUALITY - Flag to enable SCI pixel quality bits.

(enable_trcor = yes) [bool]
Tool Specific Parameter - XISPI.

XISPI - Flag to enable charge trail correction.

(enable_cticor = yes) [bool]
Tool Specific Parameter - XISPI.

XISPI - Flag to enable CTI correction.

(enable_scicti = yes) [bool]
Tool Specific Parameter - XISPI.

XISPI - Flag to enable CTI correction for SCI.

(enable_edge_smooth = yes) [bool]
Tool Specific Parameter - XISPI.

XISPI - Flag to enable smoothing the PHA to PI relation around edge.

(hk_time_margin = 3600.0) [real]
Tool Specific Parameter - XISPI.

XISPI - Time margin in second to consider AE-temp is valid.

(hk_aetemp_min = -30.0) [real]
Tool Specific Parameter - XISPI.

XISPI - Minimum value in degC to consider AE-temp is valid.

(hk_aetemp_max = 40.0) [real]
Tool Specific Parameter - XISPI.

XISPI - Maximum value in degC to consider AE-temp is valid.

(flag_rand_phas0 = yes) [bool]
Tool Specific Parameter - XISPI.

XISPI - Flag to randomize PHAS[0].

(flag_constant_spth = no) [bool]
Tool Specific Parameter - XISPI.

XISPI - Flag if the constant split threshold method is applied.

(yes: constant split threshold, no: the "energy" dependent split)

(constant_spth = 20) [int]
Tool Specific Parameter - XISPI.

XISPI - Value of the split threshold, when the constant method is selected.

(cleansis_run = yes) [bool]
Tool Specific Parameter - CLEANSIS.

CLEANSIS - Run yes or no.

(cellsize = 5) [int]
Tool Specific Parameter - CLEANSIS.

CLEANSIS - Search cell size in units of pixels.

(logprob = -5.3) [real]
Tool Specific Parameter - CLEANSIS.

CLEANSIS - The LOG of the Poisson probability threshold for rejecting a pixel.

(bthresh = 3) [int]
Tool Specific Parameter - CLEANSIS.

CLEANSIS - Zero background cutoff threshold.

(xis0_expr = DEFAULT) [string]
MAKETIME XIS0 expression. The boolean expression used to calculate good time intervals (GTI) for XIS0 screened events from the mkf file. You may use the "@filename" construction to load a complex expression from a file. The file may contain multiple lines (which are concatenated together). "DEFAULT" uses the production pipeline values:

SAA_HXD == 0 && T_SAA_HXD>436 && ELV>5 && DYE_ELV>20 && ANG_DIST<1.5 && S0_DTRATE<3 && AOCU_HK_CNT3_NML_P==1

(xis1_expr = DEFAULT) [string]
MAKETIME XIS1 expression. The boolean expression used to calculate good time intervals (GTI) for XIS1 screened events from the mkf file. See xis0_expr. "DEFAULT" uses the production pipeline values:

SAA_HXD == 0 && T_SAA_HXD>436 && ELV>5 && DYE_ELV>20 && ANG_DIST<1.5 && S1_DTRATE<3 && AOCU_HK_CNT3_NML_P==1

(xis2_expr = DEFAULT) [string]
MAKETIME XIS2 expression. The boolean expression used to calculate good time intervals (GTI) for XIS2 screened events from the mkf file. See xis0_expr. "DEFAULT" uses the production pipeline values:

SAA_HXD == 0 && T_SAA_HXD>436 && ELV>5 && DYE_ELV>20 && ANG_DIST<1.5 && S2_DTRATE<3 && AOCU_HK_CNT3_NML_P==1

(xis3_expr = DEFAULT) [string]
MAKETIME XIS3 expression. The boolean expression used to calculate good time intervals (GTI) for XIS3 screened events from the mkf file. See xis0_expr. "DEFAULT" uses the production pipeline values:

SAA_HXD == 0 && T_SAA_HXD>436 && ELV>5 && DYE_ELV>20 && ANG_DIST<1.5 && S3_DTRATE<3 && AOCU_HK_CNT3_NML_P==1

(hxd_pin_expr = DEFAULT) [string]
MAKETIME HXD PIN expression. The boolean expression used to calculate good time intervals (GTI) for HXD/PIN screened events from the mkf file. See xis0_expr. "DEFAULT" uses the production pipeline values:

SAA_HXD==0 && T_SAA_HXD>500 && TN_SAA_HXD>180 && ELV>5 && ANG_DIST<1.5 && AOCU_HK_CNT3_NML_P==1 && COR>6 && HXD_HV_W0_CAL>700 && HXD_HV_W1_CAL>700 && HXD_HV_W2_CAL>700 && HXD_HV_W3_CAL>700 && HXD_HV_T0_CAL>700 && HXD_HV_T1_CAL>700 && HXD_HV_T2_CAL>700 && HXD_HV_T3_CAL>700

NOTE: aepipeline will create "Course", "Normal" and "Fine" clock mode PIN screened event files using the criteria above (or the user-specified value for hxd_pin_expr) AND'd with the following clock mode selections:

  1. Course - HXD_WPU_CLK_RATE == 0
  2. Normal - HXD_WPU_CLK_RATE == 1
  3. Fine - HXD_WPU_CLK_RATE > 1

But the "Fine" and "Course" modes have not been used onboard Suzaku as of yet.

(hxd_gso_expr = DEFAULT) [string]
MAKETIME HXD GSO expression. The boolean expression used to calculate good time intervals (GTI) for HXD/GSO screened events from the mkf file. See xis0_expr. "DEFAULT" uses the production pipeline values:

SAA_HXD==0 && T_SAA_HXD>500 && TN_SAA_HXD>180 && ELV>5 && ANG_DIST<1.5 && HXD_DTRATE<3 && AOCU_HK_CNT3_NML_P==1 && COR>6 && HXD_HV_W0_CAL>700 && HXD_HV_W1_CAL>700 && HXD_HV_W2_CAL>700 && HXD_HV_W3_CAL>700 && HXD_HV_T0_CAL>700 && HXD_HV_T1_CAL>700 && HXD_HV_T2_CAL>700 && HXD_HV_T3_CAL>700

NOTE: aepipeline will create "Course", "Normal" and "Fine" clock mode GSO screened event files using the criteria above (or the user-specified value for hxd_gso_expr) AND'd with the following clock mode selections:

But the "Fine" and "Course" modes have not been used onboard Suzaku as of yet.

(hxd_pse_expr = DEFAULT) [string]
MAKETIME HXD PSE (pseudo event) expression. The boolean expression used to calculate good time intervals (GTI) for the HXD pseudo-event file from the mkf file. See xis0_expr. "DEFAULT" uses the production pipeline values:

SAA_HXD==0 && T_SAA_HXD>500 && TN_SAA_HXD>180 && ELV>5 && ANG_DIST<1.5 && AOCU_HK_CNT3_NML_P==1 && COR>6 && HXD_HV_W0_CAL>700 && HXD_HV_W1_CAL>700 && HXD_HV_W2_CAL>700 && HXD_HV_W3_CAL>700 && HXD_HV_T0_CAL>700 && HXD_HV_T1_CAL>700 && HXD_HV_T2_CAL>700 && HXD_HV_T3_CAL>700

NOTE: If the PSE file created with aepipeline will be used to correct for deadtime in the PIN and/or GSO screened event files, the expression for hxd_pse_expr MUST include all time covered by the PIN and/or GSO expressions (hxd_pin_expr and hxd_gso_expr parameters).

(xis0_grade = DEFAULT) [string]
EXTRACTOR XIS0 GRADE filtering expression. Only events matching this selection will be included in the screened event files. The format must be a comma delimited list of colon separated pairs of numbers, with each pair representing an inclusive range of allowed GRADE values. A value of "DEFAULT" for this parameter will use "0:0,2:4,6:6" to select events with GRADE equal to any of 0, 2, 3, 4, or 6.

(xis1_grade = DEFAULT) [string]
EXTRACTOR XIS1 GRADE filtering expression. See xis0_grade

(xis2_grade = DEFAULT) [string]
EXTRACTOR XIS2 GRADE filtering expression. See xis0_grade

(xis3_grade = DEFAULT) [string]
EXTRACTOR XIS3 GRADE filtering expression. See xis0_grade

(xis0_psum_grade = DEFAULT) [string]
EXTRACTOR XIS0 GRADE filtering expression for PSUM/Timing mode. Only events matching this selection will be included in the screened event files. The format must be a comma delimited list of colon separated pairs of numbers, with each pair representing an inclusive range of allowed GRADE values. A value of "DEFAULT" for this parameter will use "0:2" to select events with GRADE equal to any of 0, 1, 2.

(xis1_psum_grade = DEFAULT) [string]
EXTRACTOR XIS1 GRADE filtering expression for PSUM/Timing mode. See xis0_psum_grade

(xis2_psum_grade = DEFAULT) [string]
EXTRACTOR XIS2 GRADE filtering expression for PSUM/Timing mode. See xis0_psum_grade

(xis3_psum_grade = DEFAULT) [string]
EXTRACTOR XIS3 GRADE filtering expression for PSUM/Timing mode. See xis0_psum_grade

EXAMPLES

  1. The following command will recalibrate (stage 1) and re-screen (stage 2) all XIS and HXD data for sequence 100039010 that currently resides in the directory /data/100039010/, and the output will be stored in a directory called /data/100039010_reproc/:

    aepipeline indir=/data/100039010 outdir=/data/100039010_reproc entry_stage=1 exit_stage=2 steminputs=ae100039010 instrument=ALL

  2. The following command will re-screen (stage 2 only) XIS1 data for the same data set as in the previous example:

    aepipeline indir=/data/100039010 outdir=/data/100039010_reproc entry_stage=2 exit_stage=2 steminputs=ae100039010 instrument=XIS1

  3. The following command will re-screen (stage 2 only) XIS1 data for the same data set as in the previous example selecting only events with GRADE==0:

    aepipeline indir=/data/100039010 outdir=/data/100039010_reproc entry_stage=2 exit_stage=2 steminputs=ae100039010 instrument=XIS1 xis1_grade="0:0"

NOTES

None, but see help for individual parameters above.

LAST MODIFIED

April 2014