Rex is a script to make basic RXTE PCA and HEXTE data analysis more
Rex is a script designed to run through the basic data reduction of
PCA and HEXTE data for multiple observations within a given proposal. It
reads the index files in the RXTE database structure and extracts the
most commonly used data products from the PCA Standard2 or the HEXTE
Archive data for as many ObsIds as it finds in the database. In
addition to filtering the data using the GOF-recommended criteria, the
script produces model background files to subtract from the data. The
final products are a light curve and a spectrum for each individual
observation, plus an overall light curve and spectrum for the all
Note: Rex is meant to be a convenient tool for doing the same
basic analysis on a large amount of data, not as a
one-script-fits-all substitute for understanding how to use XTE data
and software. Before using Rex, familiarize yourself with your data
and the software tools that Rex calls by reading the ABC Guide and by analyzing at
least one observation by hand, start to finish, following the
and HEXTE recipes.
Rex was designed to be easy to use and fairly robust, in some ways at the expense of flexibility. The script can be run from the command line with nothing more specified than the proposal level directory of the data to be extracted. Its defaults are intended for the analysis of faint sources such as AGN, and its power lies in its convenience for large numbers of observations, especially in the case of long monitoring campaigns. Most of the defaults, however, can be changed allowing the scientist to customize the analysis without having to do it all by hand.
What it does:
The user obtains an auxiliary directory of tools (available as a small tar file on our ftp area), places it in the working directory, and runs the script as outlined below. Rex uses the contents of that "aux" directory to perform the following steps for each ObsId found in the proposal's Master Index file:
- check that it is not a slew (unless flagged -s to extract the slew data) and is the right target (if one is specified);
- run xtefilt to produce an updated filter file containing the quantities found in aux/appidlist; this filter file will be put in the proposal/ObsId/stdprod/ directory, named with the suffix ".xfl", and listed in the standard products index file (the FIST... file) for that ObsId;
- make a directory for the ObsId in both the aux directory (for file lists, GTIs, background files, etc.) and in the working directory (for resulting light curves and spectra); compile a list of Standard2 files by dumping FIPC index file;
- run pcabackest to produce modeled background data for each Standard 2 file found in the ObsId/pca/ directory's index file (the FIPC... file); use the background models listed in aux/model.files and the SAA history file there; the output background files go into the aux/ObsId directory along with a list of all files successfully created;
- run maketime to produce a list of Good Time Intervals (GTI) to be extracted; it uses the filter file and applies the criteria in aux/expression.txt; the result is checked for good time, and if no time fits those criteria, Rex goes onto the next ObsId without attempting to extract;
- run saextrct to produce spectra and/or lightcurves for source and background files (assuming GTI and background files were created successfully) using the defaults below;
- subtract the background light curve from the source using lcmath and/or writes the background spectrum into the BACKFILE keyword of the source spectrum;
Finally, Rex creates overall lightcurves and spectra for
all observations combined.
If the user is analyzing HEXTE data, the appropriate background is
assembled from the off-source pointings, and subtracted from
the data as per our online
What it makes:
The default extracted products are light curves and spectra extracted from the PCA Standard 2 data using:
and times when:
- layer 1 only, for better signal to noise;
- PCUs 0, 1, and 2, since PCUs 3 and 4 go on and off regularly;
- absolute channels 0 to 27 (light curves only), or approximately 1-10 keV for epoch 3 data; see Channel Energy Conversion Table.
These criteria are the conservative cuts the GOF and PCA team recommend to ensure the best background subtraction and highest signal to noise in the data. The scientist can change any of them, however (see below), to suit specific analysis situations. We recommend examining the data carefully to be sure than no spurious effects creep in.
- the Earth elevation angle is greater than 10 degrees (bright Earth effects have been seen near 5 degrees, thus the conservative cut);
- the pointing offset is less than 0.02 degrees;
- the time since the peak of the last SAA passage is greater than 30 minutes;
- the electron contamination is less than 0.1.
The results are named with the layer(s) extracted, the channel range (for lightcurves), and for background or net where appropriate. If a root name is specified, this will be prepended onto the default names. For example, Rex run with all defaults on the proposal P30401 will produce the following in the working directory:
layer1.pha - layer 1 spectrum from all ObsIds
individual ObsIds' products:
layer1_bkg.pha - layer 1 background spectrum from all ObsIds
layer1_ch0-27.lc - layer 1, ~1-10 keV light curve from all ObsIds
layer1_ch0-27_bkg.lc - layer 1, ~1-10 keV background light curve from all ObsIds
layer1_ch0-27_net.lc - layer 1, ~1-10 keV background subtracted light curve from all ObsIds
30401-01-01-00/ all of the above for one ObsId
tools and intermediate products added to aux directory:
30401-01-02-00/ all of the above for one ObsId
30401-01-03-00/ all of the above for one ObsId
30401-01-04-00/ all of the above for one ObsId
30401-01-05-00/ all of the above for one ObsId
aux/30401-01-01-00/FS4a_...._bkg - all background data files for this ObsId
plus log files for each extract run, maketime call, etc.
aux/30401-01-01-00/basic.gti - list of Good Time Intervals for this ObsId
aux/30401-01-01-00/bkg.xdf - time ordered list of background files for this ObsId
aux/30401-01-01-00/std2.xdf - time ordered list of Standard 2 mode data files for this ObsId
for each ObsId, etc.
For HEXTE operation, the combined products will have the generic names
hxt0_ch15-60.lc etc (for whatever channel range is chosen;
the default range is the same as for the PCA: 0-27).
Rex can be run from the command line giving only the data directory and the flag to quell prompts and take the defaults. Or it can be run by typing only "rex" and then answering the prompts one at a time.
The first step to running Rex is to retrieve and put into place the auxiliary directory of tools it will need. A tar file of all the necessary pieces can be found on our ftp area atftp://legacy.gsfc.nasa.gov/xte/software/rex/aux.tar.
Untar this file in the working directory from which you will run Rex and where you want all of the products. (This should not be anywhere under the hierarchical database structure; in general, work and results should be kept separate from the data area to avoid any possible corruption or confusion.) Check that the SAA history file (aux/pca_saa_history) covers the time span of your data by using fkeyprint to look at the CVED0001 keyword. This is the end date of the time covered by this file, so if your data fall later than this date, you must update the pca_saa_history file by getting a new one from our ftp site. It is generally updated every few weeks, but the aux.tar file may be behind.
- To run from the command line with all defaults, give the path to the proposal directory and the -q flag to quell the prompting:
% rex -d /local/data/P30408 -q
Rex will then run on the PCA data from every ObsId using all of the defaults specified above.
- To run from the command line changing the defaults, specify each parameter followed by the desired value:
% rex -d /local/data/P30408 -c 5-45 -l all -q
This will tell Rex to extract all layers and to create full spectra and light curves from channels 5-45. The last flag, -q, tells it not to prompt for target, printmode, or root name, but rather take the defaults of "all", "both", and "none", respectively.
- To be prompted for all parameters except for the flags:
Path to proposal level directory containing FMI: /local/data/P30408
Printmode, l (ligh curve), s (spectra) [both]: s
Channels for light curve extraction [0-27]:
Layers to be extracted (1,2,3, or all) : all
Root name for products : cenA
Rex will then extract spectra from all layers for all targets in the proposal directory. It will prepend the string "cenA" onto all the output light curves and spectra. In this example, rex will produce in each resulting obsid directory the files cenA_layerall.pha and cenA_layerall_bkg.pha and the same at the top level for all obsids combined.
- To bypass time consuming steps which have already been run, Rex has two hidden flags, -f and -b, which respectively cause it to skip the xtefilt and pcabackest steps.
Use -f only if you are sure that your filter files contain all columns needed by pcabackest (PCUn_ON, BKGD_THETA, BKGD_PHI) and maketime (all those listed in the selection expression in aux/expression.txt such as ELV, OFFSET, PCUn_ON, TIME_SINCE_SAA, and ELECTRON2.) If you have previously run xtefilt v1.6 or later (using the -s option to place the result in the standard products directory) with an up-to-date appidlist, you can safely use this flag to omit this step.
Use -b only if Rex has already been run successfully on all the observations you are interested in. It will put the correct file lists where it needs them. If, however, Rex had a problem with one or more background files, it will not include them in the file lists. You can add them by hand if you've fixed the problem (see below) and re-run pcabackest yourself, or run Rex again, whichever is easier in the particular case.
- To analyze HEXTE data, Rex has the hidden flag -x. This
tells Rex to analyze the HEXTE data for whichever cluster you supply
as the argument to the -x flag. For example, if you have already run
Rex once to create the filter files and perform a run on PCA Standard
2 data, your next step might be to do a HEXTE run:
% rex -f -d /local/data/P30408 -x 0 -c 15-60
This will analyze HEXTE Cluster 0 data between channels 15 and
60. (Information not given on the command line will be prompted for.) It
will skip the generation of the filter files (as these are already
made), separate out background data using HXTBACK, extract spectra and
perform the deadtime corrections using HXTDEAD, and merge the results.
The default is to use the data from all four HEXTE detectors in
the cluster chosen; note that one of the detectors in Cluster 1 no
longer provides spectral information, so you may want to explicitly
change the default when analyzing Cluster 1 data (see below).
- To change the selection expression used by maketime, you can edit the expression contained in the auxiliary file aux/expression.txt. This expression is passed directly to maketime and can take either C or Fortran syntax. Read the fhelp for maketime for more information about this tool.
This is how one would change the requirement for certain PCUs to be
on, e.g. to add a check for PCUs 3 and 4 if you have determined that
they are on for your observations and you wish to add them to the
extraction, or alternatively e.g. remove PCU 1 from the selection
expression if you've determined that this PCU was turned off during
your observation It is also where you can adjust the ELV and
TIME_SINCE_SAA to be more or less conservative, as you see fit based
on examining the data. If you want to examine slew data or occults,
you would change the ELV and/or OFFSET criteria appropriately.
- To change the number of PCUs extracted, you can add or remove columns from the lists in the auxiliary files. For instance, to add PCUs 3 and 4 to the layer 1 extraction, add the columns
to the text file aux/layer1.cols. Make sure to add a test to your maketime expression that these PCUs are on (as described above) to ensure a consistent set of data is extracted.
- To change the background model used, you can put the desired model files in the aux directory (or not), and edit the aux/model.files list to point to the new files, wherever they are. If you want to compare models, you will have to set up a separate results directory from which to run Rex (with a separate aux, of course.) Otherwise, Rex will overwrite previous background products, since all background files are simply named with "_bkg".
- To change the HEXTE detectors used, edit the
aux/hexte.cols list. For example, to exclude the HEXTE
detector in Cluster 1 that no longer provides spectral information,
you would delete one line of this file to leave:
Once you've used Rex to create source and background spectra,
source and background light curves, etc, you will probably want to do
a few more things. Before undertaking spectral analysis of PCA data, you'll
need to create a response
matrix for your spectrum using the pcarsp tool.
If you're new to X-ray data analysis, you'll want to study
the online information about the XSPEC and
XRONOS packages, for spectral and timing analysis respectively.
For further information about RXTE data and the FTOOLS suite available
for their analysis, please see
the extensive user guides on our
data analysis pages,
Getting Started Guide,
the complete guide to RXTE data in
The ABC of XTE,
and especially the recipes in the
RXTE Cook Book.
Notes and Trouble Shooting
Rex attempts to trap most common errors and will print a warning to the screen if something generates an error. The warning will include the command line and tell where to find the log file for a complete record of the tool's output. Try cutting and pasting the command which caused the error and run it from the command line prompt. Use common sense and read the fhelp for the tool in question to diagnose the problem. A few specific examples are discussed below, but considering the number of different tools this script calls, the possibilities for error are numerous. If the error message is opaque, send the log of the particular problem along with a screen dump showing your inputs to firstname.lastname@example.org.
- Not all observations in the proposal directory were extracted:
Rex reads the index files to find the data; if these index files are not accurate, Rex may fail to find everything it should. The observations should be listed in the proposal level FMI, which can be updated using the tool recofmi if it is missing ObsIds that are in the proposal directory.
Rex will check the good time intervals determined by maketime on the basis of the above criteria, and if there are no times matching these criteria for a given obsid, it will not attempt to extract any products. It will print a message to the screen to this effect when it hits such an obsid. Check the filtering criteria against the data by plotting from the filter file (the obsid/stdprod/FP*xfl file) the quantities found in aux/expression.txt. These criteria can be adjusted by editing that expression.
- How to recover from the general case of an error which causes an obsid or a single file to be skipped:
One disadvantage of having a master script to go through a lot of computation autonomously is that a small problem in one section isn't easy to fill in after the script is finished. To make diagnosing problems easier, the log file that Rex writes when there is an error stores the entire command line, and this can be used to run by hand the particular job that failed. For instance, if an individual pcabackest run needs a larger timeslop than the default 128 seconds, the user can grab the original command from the error log, change the timeslop to be large enough, and run it again. But as with any pcabackest error, that background file won't have been written to the file list for that obsid or to the master list of all background files. The user can then edit the aux/obsid/bkg.xdf file and insert the missing entry with the correct path and in correct chronological order. Then, Rex can be run with the -b option to skip the background generation, and the extraction will then include the previously missing file.
Note that if there are any problems with the background generations, they must be fixed before Rex can be run again with the -b option. The first time it is run, Rex determines whether pcabackest completed successfully before writing it to the lists to be extracted. If it is then run with the -b option, the individual obsid bkg.xdf lists remain the same with those problem files excluded, but the allbkg.xdf list is remade with the assumption that all Std2 files have corresponding, correct background files. (This is a necessary result of the ability of the script to be re-run with a different target extracted.) So any problems with those background files will be avoided in the individual obsid extractions, but will cause a problem with the combined extraction unless they are fixed by hand.
If you have a question about RXTE, please send email to one of our