NAME

sxipipeline - SXI reprocessing tool

USAGE

sxspipeline indir outdir steminputs stemoutputs entry_stage attitude orbit obsgti housekeeping

DESCRIPTION

sxipipeline duplicates most of the pipeline (not trend data) for the SXI. It allows the user to run all or part of the pipeline processing and to vary the calibration files and filtering (screening) criteria used. A number of other pipeline processing parameters can also be changed.

SXI Pipeline Stages

The sxipipeline is divided into 3 stages:

  1. Calibration
  2. Data screening
  3. Product creation
Each stage may be run singly or in combination with the preceding stages. This is controlled by the entry_stage and exit_stage parameters.

SXI Stage 1 consists of the following steps:

    For each event file:
  1. (optional) Run coordevt on hot pixel file
  2. Run coordevt
  3. Run sxiphas
  4. Run sxiflagpix
  5. Run sxipi
  6. Run sxipi
    1. Create flickering pixel file
      1. Filter SXI events STATUS[1]==b0
      2. Run searchflickpix
      3. Run coordevt on flickering pixel events
  7. Run sxiflagpix with flickering pixels

The data screening (Stage 2) is identical to that in the production pipeline, when default parameters are used. For details on the default screening applied to the SXI events (respectively), see:

Default GTI used for screening data are:

Optionally, sximodegti can be run to create new GTI specific to the DATACLASSes of the input SXI event file.

The product creation (Stage 3) is identical to that in the production pipeline, when default parameters are used. For SXI events extractor is run on SKY coordinates and a lightcurve, spectra and images are created for each cleaned event file. No gif images are created.

INPUT

The input to sxipipeline is specified using (at minimum) the indir parameter. This should be specified as the sxi event_uf level sequence directory, e.g.:

sxipipeline indir=/path/to/100039010/sxi/event_uf

Paths to specific sxi housekeeping and satellite data files can be specified using the attitude, housekeeping, extended_housekeeping, makefilter, orbit and obsgti parameters. The attitude, orbit and sxi housekeeping files are required for stage 1 calibration.

OUTPUT

Filenames, etc.

The number of output files depends on both pipeline processing stage(s) and the options selected. All output files are written to the directory specified by the outdir parameter. The archive directory structure is NOT reproduced (i.e. all output files are in a single directory).

The names of files produced are the same as those found in the HEASARC archive. However the usual "ahXXXXXXXXX" prefix, where "XXXXXXXXX" is the sequence number, can be replaced by a character string set by the stemoutputs parameter. This defaults to the value set by the steminputs parameter.

PARAMETERS

indir [string]
Directory containing the input data. This should be the path to the event_uf (or event_cl for stage_3 processing)

    /path/to/805062010/sxi/event_uf

outdir [string]
Output directory used for all output data, as well as the report file.

CAUTION: If clobber is set, and this directory already exists, then this task overwrites files as needed in this directory.

steminputs [string]
Stem for FITS input files, e.g. ah100039010. This string is used as the base filename for finding input files. For example, if steminputs=ah100039010, then to find the attitude file, ahpipeline matches the regular expression /ah100039010sxi_[ps][\d][0-9a-f]{8}_uf\.evt(\..+)?$/ against all files found in the indir directory.

(stemoutputs = DEFAULT) [string]
Base (stem) output name used for creating output files. If set to "DEFAULT", then steminputs are used.

entry_stage = 1 [1|2|3]
Entry stage, 1 or 2.

Stage 1: Re-calibrate unfiltered event files.

Stage 2: Start from existing unfiltered event files.

exit_stage = 2 [1|2|3]
Exit stage, 1 or 2.

Stage 1: Produces calibrated unfiltered event files.

Stage 2: Produces screened event files.

attitude [string]
Attitude file

(extended_housekeeping = ah1001.ehk) [string]
Extended housekeeping file

(makefilter = ah1001.mkf) [string]
Makefilter file

orbit [string]
Orbit file

gtifile [string]
Input GTI file

housekeeping [string]
SXI Housekeeping file

(regionfile = NONE) [string]
Input region file

(sxi_mkflabel = SXI#SCI) [string]
Label to use for SXI MKF GTI creation. The hash is replaced by a respective mode: CCDW1, CCD12WA or CCD34WF

(sxi_ehklabel = SXI#SCI) [string]
Label to use for SXI EHK GTI creation. The hash is replaced by a respective mode: CCDW1, CCD12WA or CCD34WF

(sxi_evtlabel = SXI#SCI) [string]
Label to use for SXI event screening. The hash is replaced by a respective mode: CCDW1, CCD12WA or CCD34WF

(sxi_start = 0.0) [real]
SXI CALDB start time

(ra = -999.99999) [real]
RA of nominal pointing [deg]

(dec = -999.99999) [real]
Dec of nominal pointing [deg]

(roll = 0.0) [real]
Roll of nominal pointing [deg]

(optdetx = -999.99999) [real]
SXI optical detx coordinate

(optdety = -999.99999) [real]
SXI optical dety coordinate

(optfocx = -999.99999) [real]
SXI optical focx coordinate

(optfocy = -999.99999) [real]
SXI optical focy coordinate

(optskyx = -999.99999) [real]
SXI optical skyx coordinate

(optskyy = -999.99999) [real]
SXI optical skyy coordinate

(ra_pnt = -999.99999) [real]
RA of sxi pointing [deg]

(dec_pnt = -999.99999) [real]
DEC of sxi pointing [deg]

(calc_hotpix = no) [boolean]
Run coordevt on hot pixel file (yes/[no])

(calc_modegti = yes) [boolean]
Calculate SXI data mode GTI ([yes]/no)

(teldeffile = CALDB) [string]
Input teldef file (or CALDB)

(leapsecfile = REFDATA) [file]
ahscreen: Input leap second file (or CALDB, [REFDATA])

(selectfile = CALDB) [file]
ahscreen: Input file with the selection expressions

(badpixfile = CALDB) [file]
sxiflagpix: badpixfile

(maskfile = CALDB) [file]
sxiflagpix: maskfile

(vtevnoddfile = CALDB) [file]
sxipi: evenodd

(ctifile = CALDB) [file]
sxipi: cti

(chtrailfile = CALDB) [file]
sxipi: chtrail

(spthfile = CALDB) [file]
sxipi: splitth

(gainfile = CALDB) [file]
sxipi: gain

(patternfile = CALDB) [file]
sxipi: grade

(dattfile = datt.out) [string]
output datt file with drift corrections

(coordevt_startsys = LOWEST) [string]
Starting coordinate system

(stopsys = HIGHEST) [string]
Final coordinate system

(annaber = no) [string]
Apply annual aberration correction (yes, [no], INVERT)

(followsun = no) [boolean]
Recalculate the Sun position for each event (yes, [no])

(orbaber = no) [string]
Apply sat orbital aberration correction (yes, [no], INVERT)

(attinterp = LINEAR) [string]
Sky attitude interpolation method (LINEAR, CONSTANT)

(dattinterp = LINEAR) [string]
Delta attitude interpolation method (LINEAR, CONSTANT)

(attdt = 32.) [real]
Allowed margin for time extrapolation in attfile [s]

(dattdt = 0.5) [real]
Allowed margin for time extrapolation in dattfile [s]

(chkattgap = no) [boolean]
Limit attitude interpolation if gaps present (yes, [no])

(chkdattgap = yes) [boolean]
Limit delta attitude interpolation if gaps present ([yes], no)

(attext = ATTITUDE) [string]
Attitude extension

(attcol = QPARAM) [string]
Attitude column

(attform = QUAT) [string]
Attitude format ([QUAT], EULER)

(orbext = ORBIT) [string]
Orbit extension

(orbcol = VELOCITY) [string]
Orbital velocity column

(orbform = VECTOR) [string]
Orbital velocity format ([VECTOR], COMPONENTS, KEPLERIAN)

(coordevt_randomize = TELDEF) [string]
Randomize coordinates when rebinning ([TELDEF], yes, no)

(randsys = TELDEF) [string]
Starting system for randomization (or TELDEF)

(randscalesys = TELDEF) [string]
System to determine randomization amount (or TELDEF)

(infileext = EVENTS) [string]
Event extension

(inclfloatcol = no) [boolean]
Write non-rounded coordinate columns (yes, [no])

(inclfloatskycol = no) [boolean]
Write non-rounded sky coordinate columns (yes, [no])

(floatcolsuffix = _FLOAT) [string]
Suffix for non-rounded coordinate columns

(startwithfloat = no) [boolean]
Start with non-rounded startsys coordinates (yes, [no])

(blankcol = yes) [boolean]
Assign null values to columns not calculated ([yes], no)

(btnull = 255) [int]
TNULL for byte (B) columns

(itnull = -999) [int]
TNULL for short (I) columns

(jtnull = -999) [int]
TNULL for long (J) columns

(ktnull = -999) [int]
TNULL for long (K) columns

(sbtnull = 255) [int]
TNULL for signed byte columns

(uitnull = -999) [int]
TNULL for unsigned short columns

(ujtnull = -999) [int]
TNULL for unsigned long columns

(colbound = -32768) [string]
TNULL, TLMIN, TLMAX for PHAS

(chipcol = CCD_ID) [string]
Chip column (or NONE)

(xcol = ACTX) [string]
X coordinate column

(ycol = ACTY) [string]
Y coordinate column

(chancol = PI) [string]
Pulse height column (or NONE)

(gradecol = GRADE) [string]
Event grade column (or NONE)

(grade = 0) [string]
Event grade for clean (or ALL)

(n_division = 1) [int]
Divide total observation time into the given number

(cleanimg = no) [boolean]
Output cleaned image for debugging (yes, no)

(cellsize = 7) [int]
Poisson clean cell size (odd integer > 1)

(impfac = 320) [double]
Factor for gamma function

(logprob1 = -5.6) [double]
Log Poisson probability threshold

(logprob2 = -5.6) [double]
Log Poisson probability threshold for second step

(iterate = yes) [boolean]
Iterate the second step Poisson clean (yes, no)

(flagedge = no) [boolean]
Zero chip edge pixels (yes, no)

(bthresh = 3) [int]
Zero background threshold

(duration = no) [boolean]
Perform detailed search for flickering duration (yes, no)

(sigma = 3.0) [double]
Significance level for flickering duration

(firstchip = TLMIN) [string]
Min value for chip number

(lastchip = TLMAX) [string]
Max value for chip number

(xmin = TLMIN) [string]
Min value for X coordinate

(xmax = TLMAX) [string]
Max value for X coordinate

(ymin = TLMIN) [string]
Min value for Y coordinate

(ymax = TLMAX) [string]
Max value for Y coordinate

(chanmin = TLMIN) [string]
Min pulse-height value for clean (inclusive)

(chanmax = TLMAX) [string]
Max pulse-height value for clean (inclusive)

(outbadpix = no) [boolean]
Output bad pixel file (yes/[no]). This parameter is not a boolean in sxiflagpix but rather a filename. For ahpipeline and sxipipeline this is boolean to account for multiple files

(outbadimg = yes) [boolean]
Output bad pixel image ([yes]/no). This parameter is not a boolean in sxiflagpix but rather a filename. For ahpipeline and sxipipeline this is boolean to account for multiple files

(npixnbr = 1) [int]
Pixel distance defining a neighbor

(nboundnbr = 1) [int]
Pixel distance defining neighbor from CCD/window/segment boundary

(citrailnbr = 2) [int]
Pixel distance trailing CI row

(ciprenbr = 1) [int]
Pixel distance preceding CI row

(echoflag = yes) [integer]
Flag CR echo pixels ([yes]/no)

(echomap = yes) [integer]
Output CR echo pixel fraction map (yes/[no])

(echonbr = 2) [integer]
Distance in pixels from a cosmic ray echo pixel to flag a neighbor pixel.

(echomin = 6) [integer]
Minimum number of events for the cosmic ray echo fraction calculation.

(echospth = 15) [integer]
Split threshold for cosmic ray echo fraction calculation.

(echofrac = 0.7) [float]
Minimum fraction of hits defining a cosmic ray echo pixel. For any pixel contained in at least 'echomin' events, if at least 'echofrac' of those events have a pulse height above 'echospth', then the pixel is considered a cosmic ray echo pixel.

(bad_status = 3:9,11,12,16:19,25:28,30,37) [string]
Bad status list, colons can used to specify a range (e.g. 1:3,5 = 1,2,3,5)

(copyphas = yes) [boolean]
Copy original PHAS before processing ([yes]/no)

(resetflags = yes) [boolean]
Reset all sxiflagpix STATUS flags ([yes]/no)

(hkext = HK_SXI_USR_USER_HK1) [string]
HK extension with video temperatures

(hkcolstem = SXI_USR_HKTBL_) [string]
Column name stem for video temperatures

(hkvideoid = A,B,B,B) [string]
Video card ID for gain correction of CCD1-4

(startcol = PHAS) [string]
Starting point of correction

(evnoddcor = yes) [boolean]
Enable even-odd correction [yes/no]

(chtrailcor = yes) [boolean]
Enable charge trail correction [yes/no]

(cticor = yes) [boolean]
Enable CTI correction [yes/no]

(gaincor = yes) [boolean]
Enable gain correction [yes/no]

(ctigrade = no) [boolean]
Use grade information in CTI correction [yes/no]

(copygrade = no) [boolean]
Copy existing GRADE and PHA columns [yes/no]

(phcut = CALDB) [string]
Pulseheight cut for CTI correction, or CALDB

(badpixopt = 2) [int]
Options for events with bad pixels: ignore bad pixels (1), null bad pixels (2), null whole event (3)

(spthiter = yes) [boolean]
Enable split threshold iteration [yes/no]

(spthcaldb = yes) [boolean]
Use split thresholds from spthfile [yes/no]

(spthoffset = 15.) [real]
Split threshold offset value (if spthcaldb = no)

(spthslope = 0.) [real]
Split threshold slope value (if spthcaldb = no)

(evtthre = DEFAULT) [string]
Event threshold (or DEFAULT)

(negthre = -5000) [int]
Minimum PHAS value for normal event

(deltatime = 8) [int]
Max allowed time gap in HK temp search [s]

(debugcol = no) [boolean]
Write out the debug columns [yes/no]

(randomize = yes) [boolean]
Allow randomization in PI to UPI conversion (yes, no)

(timecol = TIME) [string]
Time column

(seed = 0) [int]
Random number generator seed (0=use system time)

(stemreport = ) [string]
File stem for log and temporary files. If the parameter is not set the, script automatically sets the stem to "sxipipeline_YYYYMMDDTHHMMSS_" and appends log file and temp file names as needed. Intended to be set by ahpipeline.

(numerrs = 0) [string]
Number of errors from sxipipeline (output)

(cleanup = yes) [boolean]
Delete temporary files ([yes]/no)

(clobber = no) [boolean]
Overwrites the existing output file if set to yes (yes/[no]).

(chatter = 2) [integer]
Chatter level for output. Set to 0 to suppress output, or to 1, 2, or 3 for increasing the chatter of the output.

(logfile = !DEFAULT) [string]
Log filename. If set to DEFAULT uses the name of the task and, if preceded by '!', overwrite the file if it exists. If set to NONE no log file is created.

(debug = no) [boolean]
Diagnostic output is printed out on the screen if set to yes (yes/[no]).

(history = yes) [boolean]
Records tool parameters in HISTORY ([yes]/no).

(mode = ql) [string ql|hl|q]
Mode to query the parameter file. Acceptable values include: 'ql' (query and learn/remember), 'hl' (hidden and learn/remember), 'q' (query but don't remember)\ , 'h' (hidden).

EXAMPLES

  1. The following command recalibrates (stage 1) and re-screens (stage 2) all SXI data for sequence 100039010 that currently resides in the directory /data/100039010/sxi/event_uf, and the output is stored in a directory called /data/100039010_reproc/:

    sxipipeline indir=/data/100039010/sxi/event_uf outdir=/data/100039010_reproc entry_stage=1 exit_stage=2 steminputs=ah100039010 attitude=/data/100039010/auxil/ah100039010/ah100039010.att orbit=/data/100039010/auxil/ah100039010/ah100039010.orb obsgti=/data/100039010/auxil/ah100039010/ah100039010_gen.gti housekeeping=/data/100039010/sxi/hk/ah100039010sxi_a0.hk makefilter=/data/100039010/auxil/ah100039010/ah100039010.mkf exended_housekeeping=/data/100039010/auxil/ah100039010/ah100039010.ehk

  2. The following command re-screens (stage 2 only) SXI data for the same data set as in the previous example, as well as recalculate the SXI mode GTI:

    sxipipeline indir=/data/100039010/sxi/event_uf outdir=/data/100039010_reproc entry_stage=2 exit_stage=2 steminputs=ah100039010 calc_modegti=yes obsgti=/data/100039010/auxil/ah100039010/ah100039010_gen.gti makefilter=/data/100039010/auxil/ah100039010/ah100039010.mkf exended_housekeeping=/data/100039010/auxil/ah100039010/ah100039010.ehk

  3. The following command creates products (stage 3 only) SXI data for a calibrated data set:

    sxipipeline indir=/data/100039010/sxi/event_cl outdir=/data/100039010_reproc entry_stage=2 exit_stage=2 steminputs=ah100039010 regionfile=none

NOTES

None, but see help for individual parameters above.

SEE ALSO

ahcalctime ahpipeline sxiphas sxiflagpix sxipi searchflickpix coordevt sximodegti ahfilter ahgtigen ahscreen

LAST MODIFIED

February 2016