NAME

sxspipeline - SXS reprocessing tool

USAGE

sxspipeline indir outdir steminputs stemoutputs entry_stage attitude orbit obsgti housekeeping timfile

DESCRIPTION

sxspipeline duplicates most of the pipeline (not trend data) for the SXS. It allows the user to run all or part of the pipeline processing and to vary the calibration files and filtering (screening) criteria used. A number of other pipeline processing parameters can also be changed.

SXS Pipeline Stages

The sxspipeline is divided into 3 stages:

  1. Calibration
  2. Data screening
  3. Product creation
Each stage may be run singly or in combination with the preceding stages. This is controlled by the entry_stage and exit_stage parameters.

SXS Stage 1 consists of the following steps:

  1. (optional) Calculate GTIFOUNDALL (gtiinvert on GTILOST)
  2. Run mxsgti
  3. Calculate <gaingti> GTI file:
  4. Run sxsanticopi
  5. For each event file:
    1. Run coordevt
    2. Run sxsflagpix
    3. Run sxssecid
  6. Calculate GHF
    1. Merge pointing and slew event files
    2. Select merged file
      • if linetocorrect eq 'MnKa':
        PIXEL==12&&ITYPE<5&&gtifilter("<gaingti>")
      • else:
        PIXEL!=12&&ITYPE<5&&gtifilter("<gaingti>")
    3. Run ftsort on filtered merged file
    4. Run sxsgain on filtered, sorted merged file
  7. For each event file:
    1. Run sxspha2pi (GHF input)
    2. Run sxsflagpix
    3. Run sxssecid
    4. Run sxsseccor
    5. Run sxspha2pi (GHF input)
    6. Run sxsperseus

By default, linetocorrect is 'MnKa' and the GHF is calculated using the 'Cal-pix' method.
The possible options for linetocorrect are:

The data screening (Stage 2) is identical to that in the production pipeline, when default parameters are used. For details on the default screening applied to the SXS events (respectively), see:

Default GTI used for screening data are:

The product creation (Stage 3) is identical to that in the production pipeline, when default parameters are used. For SXS events extractor is run on SKY coordinates and a lightcurve, spectra and images are created for each cleaned event file. No gif images are created.

INPUT

The input to sxspipeline is specified using (at minimum) the indir parameter. This should be specified as the sxs event_uf level sequence directory, e.g.:

sxspipeline indir=/path/to/100039010/sxs/event_uf

Paths to specific sxs housekeeping and satellite data files can be specified using the attitude, housekeeping, extended_housekeeping, makefilter, orbit and obsgti parameters. The attitude, orbit and sxs housekeeping files are required for stage 1 calibration.

OUTPUT

Filenames, etc.

The number of output files depends on both pipeline processing stage(s) and the options selected. All output files are written to the directory specified by the outdir parameter. The archive directory structure is NOT reproduced (i.e. all output files are in a single directory).

The names of files produced are the same as those found in the HEASARC archive. However the usual "ahXXXXXXXXX" prefix, where "XXXXXXXXX" is the sequence number, can be replaced by a character string set by the stemoutputs parameter. This defaults to the value set by the steminputs parameter.

PARAMETERS

indir [string]
Directory containing the input data. This should be the path to the event_uf (or event_cl for stage_3 processing)

    /path/to/805062010/sxs/event_uf

outdir [string]
Output directory used for all output data, as well as the report file.

CAUTION: If clobber is set, and this directory already exists, then this task overwrites files as needed in this directory.

steminputs [string]
Stem for FITS input files, e.g. ah100039010. This string is used as the base filename for finding input files. For example, if steminputs=ah100039010, then to find the attitude file, ahpipeline matches the regular expression /ah100039010sxs_[ps][\d][0-9]{4}_uf\.evt(\..+)?$/ against all files found in the indir directory.

(stemoutputs = DEFAULT) [string]
Base (stem) output name used for creating output files. If set to "DEFAULT", then steminputs is used.

entry_stage = 1 [1|2|3]
Entry stage, 1 or 2.

Stage 1: Re-calibrate unfiltered event files.

Stage 2: Start from existing unfiltered event files.

exit_stage = 2 [1|2|3]
Exit stage, 1 or 2.

Stage 1: Produces calibrated unfiltered event files.

Stage 2: Produces screened event files.

attitude [string]
Attitude file

(extended_housekeeping = ah1001.ehk) [string]
Extended housekeeping file

(makefilter = ah1001.mkf) [string]
Makefilter file

orbit [string]
Orbit file

obsgti [string]
Observation GTI file

housekeeping [string]
SXS Housekeeping file

timfile [string]
Input TIM file

(sxs_mkflabel = PIXELALL3) [string]
Label to use for SXS MKF GTI creation

(sxs_ehklabel = PIXELALL3) [string]
Label to use for SXS EHK GTI creation

(sxs_evtlabel = PIXELALL3) [string]
Label to use for SXS event screening

(regionfile = NONE) [string]
Input region file

(sxs_start = 0.0) [real]
SXS CALDB start time

(ra = -999.99999) [real]
RA of nominal pointing [deg]

(dec = -999.99999) [real]
Dec of nominal pointing [deg]

(roll = 0.0) [real]
Roll of nominal pointing [deg]

(optdetx = -999.99999) [real]
SXS optical detx coordinate

(optdety = -999.99999) [real]
SXS optical dety coordinate

(optfocx = -999.99999) [real]
SXS optical focx coordinate

(optfocy = -999.99999) [real]
SXS optical focy coordinate

(optskyx = -999.99999) [real]
SXS optical skyx coordinate

(optskyy = -999.99999) [real]
SXS optical skyy coordinate

(ra_pnt = -999.99999) [real]
RA of sxs pointing [deg]

(dec_pnt = -999.99999) [real]
DEC of sxs pointing [deg]

(calc_gtilost = no) [boolean]
Calculate SXS lost off GTI (yes/[no])

(screenlost = no) [boolean]
Screen lost events (yes/[no])

(teldeffile = CALDB) [string]
Input teldef file (or CALDB)

(leapsecfile = REFDATA) [string]
Input leap second file (or CALDB, [REFDATA])

(selectfile = CALDB) [filename]
ahscreen: Input file with the selection expressions

(gainantfile = CALDB) [filename]
Input antico gain file (or CALDB)

(pixdeffile = CALDB) [string]
Input SXS electrical pixel map file (or CALDB)

(gainfile = CALDB) [filename]
Input gain file (or CALDB)

(linefitfile = CALDB) [filename]
Input calibration line file (or CALDB)

(delayfile = CALDB) [filename]
Input instrument delay file (or CALDB)

(sxs_pulsefile = CALDB) [filename]
Input file with pulse amplitudes (or CALDB)

(adrgti = REFDATA) [string]
Input ADR GTI file (or [REFDATA])

(acphaoffset = 0.0) [float]
Average offset added to PHA values before applying the gain. A random offset is added to each PHA between -0.5+acphaoffset and +0.5+acphaoffset. So, when acphaoffset=0.5, the random offset is between 0 and 1.

(pxphaoffset = 0.0) [float]
Average offset added to PHA values before applying the gain. A random offset is added to each PHA between -0.5+acphaoffset and +0.5+acphaoffset. So, when acphaoffset=0.5, the random offset is between 0 and 1.

(stimecol = S_TIME) [string]
Name of S_TIME column

(tioncol = FWE_TI_LED#_ON) [string]
Input TI columns with LED on (#=1-4)

(tioffcol = FWE_TI_LED#_OFF) [string]
Input TI columns with LED off (#=1-4)

(plslencol = FWE_LED#_PLS_LEN) [string]
Input pulse length columns (#=1-4)

(plsspccol = FWE_LED#_PLS_SPC) [string]
Input pulse spacing columns (#=1-4)

(timeoncol = TIME_LED#_ON) [string]
Output LED-on time columns (#=1-4)

(timeoffcol = TIME_LED#_OFF) [string]
Output LED-off time columns (#=1-4)

(calctime = yes) [boolean]
Perform time assignment ([yes]/no)

(calcgti = yes) [boolean]
Produce GTI files ([yes]/no)

(afterglow = no) [boolean]
Add afterglow to fine GTI STOP times (no/[yes])

(dtdecay = CALDB) [string]
Afterglow time [s] (or CALDB)

(interp = twopoint) [string]
Interpolation method (NEAREST, TWOPOINT)

(margingti = yes) [boolean]
Create GTI between TSTART/TSTOP and first/last input GTI

(tstart = DEFAULT) [string]
Value to use for TSTART in seconds (or take from infile)

(tstop = DEFAULT) [string]
Value to use for TSTOP in seconds(or take from infile)

(dt = 0.) [real]
Time separation between input and output GTI (seconds)

(dattfile = datt.out) [string]
output datt file with drift corrections

(coordevt_startsys = LOWEST) [string]
Starting coordinate system

(stopsys = HIGHEST) [string]
Final coordinate system

(annaber = no) [string]
Apply annual aberration correction (yes, [no], INVERT)

(followsun = no) [boolean]
Recalculate the Sun position for each event (yes, [no])

(orbaber = no) [string]
Apply sat orbital aberration correction (yes, [no], INVERT)

(attinterp = LINEAR) [string]
Sky attitude interpolation method (LINEAR, CONSTANT)

(dattinterp = LINEAR) [string]
Delta attitude interpolation method (LINEAR, CONSTANT)

(attdt = 32.) [real]
Allowed margin for time extrapolation in attfile [s]

(dattdt = 0.5) [real]
Allowed margin for time extrapolation in dattfile [s]

(chkattgap = no) [boolean]
Limit attitude interpolation if gaps present (yes, [no])

(chkdattgap = yes) [boolean]
Limit delta attitude interpolation if gaps present ([yes], no)

(attext = ATTITUDE) [string]
Attitude extension

(attcol = QPARAM) [string]
Attitude column

(attform = QUAT) [string]
Attitude format ([QUAT], EULER)

(orbext = ORBIT) [string]
Orbit extension

(orbcol = VELOCITY) [string]
Orbital velocity column

(orbform = VECTOR) [string]
Orbital velocity format ([VECTOR], COMPONENTS, KEPLERIAN)

(coordevt_randomize = TELDEF) [string]
Randomize coordinates when rebinning ([TELDEF], yes, no)

(randsys = TELDEF) [string]
Starting system for randomization (or TELDEF)

(randscalesys = TELDEF) [string]
System to determine randomization amount (or TELDEF)

(infileext = EVENTS) [string]
Event extension

(timecol = TIME) [string]
Time column

(inclfloatcol = no) [boolean]
Write non-rounded coordinate columns (yes, [no])

(inclfloatskycol = no) [boolean]
Write non-rounded sky coordinate columns (yes, [no])

(floatcolsuffix = _FLOAT) [string]
Suffix for non-rounded coordinate columns

(startwithfloat = no) [boolean]
Start with non-rounded startsys coordinates (yes, [no])

(blankcol = yes) [boolean]
Assign null values to columns not calculated ([yes], no)

(btnull = 255) [integer]
TNULL for byte (B) columns

(itnull = -999) [integer]
TNULL for short (I) columns

(jtnull = -999) [integer]
TNULL for long (J) columns

(ktnull = -999) [integer]
TNULL for long (K) columns

(sbtnull = 255) [integer]
TNULL for signed byte columns

(uitnull = -999) [integer]
TNULL for unsigned short columns

(ujtnull = -999) [integer]
TNULL for unsigned long columns

(antpsp = A) [string]
Antico PSP to use for coincidence (A=PSPA B=PSPB)

(antshift = CALDB) [string]
Time shift [s] to apply to antico events (or CALDB)

(calcant = yes) [boolean]
Flag antico events ([yes]/no)

(antdtpre = CALDB) [string]
Delta time [s] preceding an antico event (or CALDB)

(antdtfol = CALDB) [string]
Delta time [s] following an antico event (or CALDB)

(antswitch = 1) [integer]
If =1 use antdtfol, =0 read delta-time from file

(antphathr = 71) [integer]
PHA threshold for antico events

(antdurthr = 2) [integer]
DURATION threshold for antico events

(calcctrec = no) [boolean]
Flag recoil cross-talk ([yes]/no)

(ctrecdt = CALDB) [string]
Delta time [s] for flagging recoil cross-talk (or CALDB)

(calcprox = yes) [boolean]
Flag electrical cross talk ([yes]/no)

(proxdt = CALDB) [string]
Delta time [s] to define simultaneous events (or CALDB)

(calcctel = yes) [boolean]
Flag electrical cross talk ([yes]/no)

(cteldt = CALDB) [string]
Delta time [s] for flagging electrical cross-talk (or CALDB)

(ctelnear = 1) [integer]
Number of pixels for flagging electrical cross-talk

(calcctel2 = yes) [boolean]
Flag electrical cross talk 2 ([yes]/no)

(cteldt2 = CALDB) [string]
Delta time [s] for flagging electrical cross-talk 2 (or CALDB)

(ctelnear2 = 1) [integer]
Number of pixels for flagging electrical cross-talk 2

(pxpithr = 600) [integer]
Events with PI values below this threshold are excluded from flagging checks given by the usepxpithr parameter.

(usepxpithr = ALL) [string]
A comma-delimited list specifying which flagging types should use the pxpithr parameter for excluding events. Allowed values in the list are ALL, NONE, PROX (proximity), CTEL (electrical cross talk), CTEL2 (2nd electrical cross talk), and CTREC (recoil cross talk). Events that do not belong to the types specified in the list are excluded from flagging regardless of their PI value.

(calcmxs = yes) [boolean]
Flag MXS pixels ([yes]/no)

(mxsdt = CALDB) [string]
Delta time [s] to extend MXS stop time (or CALDB)

(kalow = 5860.) [real]
Lower energy limit of Mn K-alpha for recoil PHA test [eV]

(kahigh = 5930.) [real]
Upper energy limit of Mn K-alpha for recoil PHA test [eV]

(kbeta = 6450.) [real]
Energy of Mn K-beta for recoil PHA test [eV]

(dtflag = no) [boolean]
Add delta-time columns for cross-talk and antico (yes/[no])

(dtprimary = CALDB) [string]
Time interval [ms] for primary (or CALDB)

(dtlowmid = CALDB) [string]
Upper time range [ms] for low secondary (or CALDB)

(dtmidhigh = CALDB) [string]
Upper time range [ms] for mid secondary (or CALDB)

(tol = 2.) [real]
Tolerance of time intervals [ns]

(regrade = no) [boolean]
Recalculate grade assignment (yes/[no])

(sxs_resetflags = yes) [boolean]
Reset all sxsflagpix STATUS flags ([yes]/no)

(phaout = PHA2) [filename]
Name of output PHA column

(gaincoeff = H) [string]
Type of gain coefficients to use ([H]/M/L)

(linetocorrect = Mnka) [string]
Line to fit (HDU name in linefitfile)

(numevent = 250) [integer]
Maximum number of events in a single spectrum

(minevent = 150) [integer]
Minimum number of events in a single spectrum

(grpoverlap = 0.) [real]
Percentage of overlap between adjacent groups

(startenergy = -1.) [real]
Start energy [eV] of bin mesh (-1 = automatic)

(stopenergy = -1.) [real]
Stop energy [eV] of bin mesh (-1 = automatic)

(extraspread = 100.) [real]
Extend bin mesh energy range [eV]

(broadening = 1.0) [real]
FWHM Gaussian broadening of calibration line profile [eV]

(gridprofile = no) [boolean]
Calculate only the grid profile (yes/[no])

(fitwidth = yes) [boolean]
Fit spectrum width (yes/[no])

(background = CONST) [string]
Fitted background type (NONE, CONST, SLOPE)

(spangti = no) [boolean]
Ignore GTI boundaries when binning spectra (yes/[no])

(usemp = no) [boolean]
Include Mp events when fitting (yes/[no])

(calcerr = no) [boolean]
Compute uncertainties on shift and width (yes/[no])

(writeerrfunc = no) [boolean]
Output uncertainty functions (yes/[no])

(avgwinrad = 30) [real]
Radius of interval [binwidth] used to update average

(minwidth0 = 1.0) [real]
Smallest allowed initial value in width fitting [binwidth]

(maxitcycle = 5) [integer]
Maximum number of fitting iterations

(r2tol = .001) [real]
Convergence criterion for R^2

(searchstepshift = 2.) [real]
Step size when fitting shift [binwidth]

(maxdshift = 5.) [real]
Largest allowed deviation from initial guess of shift [binwidth]

(bisectolshift = .001) [real]
Tolerance of shift to stop bisection method [binwidth]

(searchstepwidth = 5.) [real]
Step size when fitting width [binwidth]

(maxdwidth = 10.) [real]
Largest allowed deviation from initial guess of width [binwidth]

(bisectolwidth = .001) [real]
Tolerance of width to stop bisection method [binwidth]

(minwidth = .5) [real]
Smallest width to allow in width fitting [binwidth]

(nerrshift = 100) [integer]
Number of shift values in uncertainty calculations

(nerrwidth = 100) [integer]
Number of width values in uncertainty calculations

(shifterrfac = 3.0) [real]
Factor for determining domain of shift uncertainty arrays

(widtherrfac = 4.0) [real]
Factor for determining domain of width uncertainty arrays

(calcupi = yes) [boolean]
Calculate UPI column ([yes]/no)

(scalefile = CALDB) [filename]
Input EPI scale file for cal-pix (or CALDB)

(secphacol = PHA) [string]
Input PHA column to use for secondary correction

(scaleepi = no) [boolean]
Scale EPI values using scalefile (yes/[no])

(scalegrade = 0) [string]
List of grades to apply scale factors

(calcpi = yes) [boolean]
Calculate PI column ([yes]/no)

(addepicol = EPI2) [string]
Output energy column with secondary correction

(method = FIT) [string]
Correction method (FIT or AVERAGE)

(extended = no) [boolean]
Use extended energy range (yes/[no])

(binwidth = 0.5) [real]
PI bin width for extended energy range [eV]

(offset = 0.5) [real]
Offset for first PI for extended energy range [eV]

(tlmax = 32767) [integer]
Maximum PI channel for extended energy range

(writetemp = no) [boolean]
Output temperature used for each event (yes/[no])

(dgfile = REFDATA) [file]
Input gain coefficients file

(offsetfile = REFDATA) [file]
calibration offset file

(outrange = NULL) [file]
How events are handled outside time range

(itypecol = ITYPE) [string]
ITYPE column

(ckctrec = no) [boolean]
Exclude events with recoil cross-talk (yes/[no])

(ckctel = no) [boolean]
Exclude events with electrical cross-talk (yes/[no])

(ckant = no) [boolean]
Exclude events with antico coincidence (yes/[no])

(ckrisetime = yes) [boolean]
Do not use events with RISE_TIME > 127 ([yes]/no)

(tempidx = 2) [integer]
Input temperature index for selecting gain

(ntemp = 3) [integer]
Number of temperatures from gain file to use in interpolation

(gapdt = -1.) [real]
Time [s] between events to define a gap (or <0)

(extrap = no) [boolean]
Allow extrapolation when determining drift temperature (yes/[no])

(randomize = yes) [boolean]
Allow randomization (yes, no)

(seed = 0) [integer]
Random number generator seed (0=use system time)

(stemreport = ) [string]
File stem for log and temporary files. If the parameter is not set the script automatically sets the stem to "sxspipeline_YYYYMMDDTHHMMSS_" and appends log file and temp file names as needed. Intended to be set by ahpipeline.

(numerrs = 0) [string]
Number of errors from sxspipeline (output)

(cleanup = yes) [boolean]
Delete temporary files ([yes]/no)

(clobber = no) [boolean]
Overwrites the existing output file if set to yes (yes/[no]).

(chatter = 1) [integer]
Chatter level for output. Set to 0 to suppress output, or to 1, 2, or 3 for increasing the chatter of the output.

(logfile = !DEFAULT) [string]
Log filename. If set to DEFAULT uses the name of the task and, if preceded by '!', overwrite the file if it exists. If set to NONE no log file is created.

(debug = no) [boolean]
Diagnostic output is printed out on the screen if set to yes (yes/[no]).

(history = yes) [boolean]
Records tool parameters in HISTORY ([yes]/no).

(mode = ql) [string ql|hl|q]
Mode to query the parameter file. Acceptable values include: 'ql' (query and learn/remember), 'hl' (hidden and learn/remember), 'q' (query but don't remember)\ , 'h' (hidden).

EXAMPLES

  1. The following command recalibrates (stage 1) and re-screens (stage 2) all SXS data for sequence 100039010 that currently resides in the directory /data/100039010/sxs/event_uf, and the output is stored in a directory called /data/100039010_reproc/, as well as recalculate the SXS lost off GTI:

    sxspipeline indir=/data/100039010/sxs/event_uf outdir=/data/100039010_reproc entry_stage=1 exit_stage=2 steminputs=ah100039010 attitude=/data/100039010/auxil/ah100039010/ah100039010.att orbit=/data/100039010/auxil/ah100039010/ah100039010.orb obsgti=/data/100039010/auxil/ah100039010/ah100039010_gen.gti housekeeping=/data/100039010/sxs/hk/ah100039010sxs_a0.hk1 makefilter=/data/100039010/auxil/ah100039010/ah100039010.mkf exended_housekeeping=/data/100039010/auxil/ah100039010/ah100039010.ehk calc_gtilost=yes

  2. The following command re-screens (stage 2 only) SXS data for the same data set as in the previous example:

    sxspipeline indir=/data/100039010/sxs/event_uf outdir=/data/100039010_reproc entry_stage=2 exit_stage=2 steminputs=ah100039010 obsgti=/data/100039010/auxil/ah100039010/ah100039010_gen.gti makefilter=/data/100039010/auxil/ah100039010/ah100039010.mkf exended_housekeeping=/data/100039010/auxil/ah100039010/ah100039010.ehk

  3. The following command creates products (stage 3 only) SXS data for a calibrated data set:

    sxspipeline indir=/data/100039010/sxs/event_cl outdir=/data/100039010_reproc entry_stage=2 exit_stage=2 steminputs=ah100039010 regionfile=none

NOTES

None, but see help for individual parameters above.

SEE ALSO

ahcalctime ahpipeline mxsgti sxsanticopi coordevt sxssecid sxsflagpix sxsgain sxspha2pi ahfilter ahgtigen ahscreen

LAST MODIFIED

February 2016