This Legacy journal article was published in Volume 7, June 1998, and has not been
updated since publication. Please use the search facility above to find regularly-updated information about
this topic elsewhere on the HEASARC site.
The HEASARC RXTE Data Archive
RXTE Guest Observer Facility
The Rossi X-ray Timing Explorer (RXTE) Archive contains a large and
growing amount of high-quality timing and broad-band spectral data from a
variety of X-ray sources. This article provides an introduction to the
Proportional Counter Array (PCA), High-Energy X-ray Timing Experiment
(HEXTE), and All-Sky Monitor (ASM) instruments on board RXTE, and a
userguide to the Archive. We describe the data structure; provide
step-by-step instructions on how to browse and retrieve RXTE data using
W3BROWSE, anonymous FTP, or XDF; and give pointers for data analysis.
1. Introduction: The Rossi X-ray Timing Explorer
The RXTE satellite was launched on December 30, 1995 with the primary
objective of studying the structure and dynamics of compact X-ray sources,
including neutron star and black hole systems both within and beyond our
Galaxy. RXTE was designed to study the intensity variations of these
objects over timescales from microseconds to years. Significant
discoveries have already been made using RXTE data on both ends of this
range of timescales, including the discoveries of millisecond pulsations
in X-ray bursts, kiloHertz quasiperiodic oscillations (QPOs) in low-mass
X-ray binaries, and various flavors of long-term variability in LMXBs and
quasars. In addition to the advanced timing capabilities that give the
satellite its name, RXTE has two other critical strengths of great
importance to X-ray and multiwavelength astrophysics: its broad-band
energy response, and its unequaled observing flexibility.
RXTE carries three scientific instruments. The PCA covers the 2-90 keV
energy range, and consists of five identical collimated multi-anode
Proportional Counter Units (PCUs). The total effective area of the PCA at
the peak of its efficiency curve is approximately 7000 cm2,
making it the largest proportional counter yet flown. The detectors have
an 18% energy resolution at 6 keV and 255-channel pulse-height
discrimination. The PCA was built at NASA/GSFC.
The HEXTE consists of two independent clusters of detectors each
containing four NaI/CsI phoswich scintillation counters, covering the
energy range 15-250 keV with an intrinsic spectral resolution better than
18% at 60 keV. The two clusters contain mutually orthogonal rocking
mechanisms, which can be moved independently to provide near-simultaneous
measurements of the internal and cosmic X-ray background up to 3 degrees
on either side of the source. The PCA and the HEXTE are co-aligned with
fields of view of 1 degree. The HEXTE was designed and built at the
Center for Astrophysics & Space Sciences at the University of San
The ASM alerts scientists to flares and changes of state in X-ray
sources, and produces long-term intensity histories of bright X-ray
sources. It consists of three rotating Scanning Shadow Cameras that can
scan about 80% of the sky every 90 minutes. The cameras can also provide
positions accurate to ~3' for bright transients. The cameras are
sensitive in the 2-10 keV range, and were built at MIT. The data from the
PCA and ASM pass through the Experiment Data System (EDS, also built at
MIT) for preliminary processing.
The PCA and HEXTE data from the one-month In-orbit Checkout (IOC) phase
in January 1996 have already been made public in the RXTE Archive (~8 GB
of data), along with the pointed data from some non-proprietary Target of
Opportunity programs (24 GB at the time of writing), and the slews to and
from all targets, which are also non-proprietary (24 GB). The ASM data
are made publicly available as soon as they are processed. The
above-listed datasets have already been transferred to the RXTE Archive at
In addition to these holdings, in March 1998 data began entering the
public domain from observations performed under the RXTE Guest Observer
program. The standard proprietary period for RXTE data obtained in
response to accepted GO proposals is twelve months from the date of
receipt of the data by the PI. However, due to reprocessing activities,
the proprietary period for AO-1 data taken before December 16, 1996 has
been extended by NASA Headquarters until six months after the receipt of
the reprocessed data tape by the PI. By the end of 1998, a total of 120
GB of GO data will have been made public. After all the data from AO-1
through AO-3 have turned public, the grand total will exceed 300 GB.
In Section 2 of this article we describe the types of data available
from the PCA and HEXTE instruments, and the structure of these data in the
archive. Section 3 contains a primer on how to search the databases and
retrieve datasets of interest. Data analysis tools and documentation are
also described. In Section 4, we provide information about the available
resources for obtaining, interpreting, and analyzing data from the All-Sky
2. RXTE Data
2.1 What types of data are available?
The data from the PCA can be binned and telemetered in six different
modes simultaneously by the independent Event Analyzers (EAs) in the EDS
system. Two of these EAs are reserved for two standard PCA modes, with
timing and spectral parameters that will remain unchanged throughout the
mission to provide a uniform mission data bank. The two modes are a time
series mode, Standard 1, with 0.125-sec temporal resolution and no energy
resolution, and a spectral mode, Standard 2, with 128 channels of spectral
information and a 16-second temporal resolution. Users typically begin
their analysis by examining the Standard 2 data to gain an overview of the
observation; in fact, for many applications the Standard 2 data may be
sufficient for the user's needs.
The other four available EDS configurations were chosen by the
Principal Investigator of the original observation, generally with the
goal of maximizing the temporal and spectral information obtained by the
PCA, while staying within the constraints of the permitted telemetry
bandwidth. For fainter sources, the "Good_Xenon" modes are usually
chosen. Good Xenon data use up two of the four available EAs, and when
combined into an event list, provide the full 256-channel resolution of
the PCA instrument for all events that survive background rejection, with
event time-tagging accurate to 0.95 microseconds. Each event is also
tagged with detector and layer information, enabling the most optimal
background subtraction methods.
For brighter sources, data may have been obtained in a binned mode, a
single-bit mode (for very high time resolution data), an event mode, or
some combination of these. For full details on exactly how each of the
possible modes is defined we refer the user to the complete userguide on
the RXTE GOF Website, or to the Technical Appendix to the Announcements of
Opportunity for observing time. However, one important fact to be
absolutely clear on is that these data streams are received in parallel --
the same events can be telemetered to ground in up to six different ways,
depending upon the configurations chosen by the proposer in consultation
with the GOF or SOF.
The HEXTE has a similarly large range of possible configurations,
including an Archival mode that runs continuously, accompanied by a
user-selected mode. The vast majority of HEXTE observations are now
performed in the mode known as E_8us_256_DX1f, which is an event mode with
256-channel resolution and an 8 microsecond time resolution.
The RXTE Archive currently contains data products for each observation.
These are: a PCA light curve and spectrum from Standard 2 data, a HEXTE
light curve and spectrum, and a filter file containing a compendium of
collected and derived parameters of importance for screening, background
subtraction and data analysis (pointing position, elevation angle, number
of PCUs on, etc). The light curves and spectra were derived automatically
with a very rudimentary screening algorithm, and no background
subtraction. This renders them of some illustrative value for bright
sources, and of very little use for fainter targets. Once the Archive
matures and becomes established, a high priority for future work is the
creation of more advanced and reliable data products as a convenience to
2.2 How are the data organized in the Archive?
Archival RXTE data occupy the xte/data/archive directory on the
anonymous FTP server at legacy.gsfc.nasa.gov. Directories at the next
level down are named AO0 for IOC data, AO1 for AO-1 data, etc. Note that
you will see directories for data that are not yet public. The links are
there, but the data files themselves will not be accessible to you until
the proprietary period expires.
Below this level, RXTE data files are arranged in a hierarchical set of
directories identical in structure to the arrangement of files on a GO's
data tape. First come directories named for the proposal number (e.g.
"P10066"); the next level contains the individual observations
("ObsID's"), while the final level contains directories for each of the
Subsystems (PCA, HEXTE, ACS etc.). In the catalogue of publicly available
RXTE data, individual entries correspond to the ObsID level.
2.2.2 Observation IDs
Since the ObsID is the link from the browseable catalogue to the data
themselves, it is worth explaining its nomenclature. Each ObsID
corresponds to a single observation, where "observation" refers to a
temporally contiguous collection of data from a single pointing. The
format for the ObsIDs is as follows:
1. NNNNN is the five-digit proposal number assigned by the
GOF, identical to that contained in the name of the
2. TT is the two-digit target number assigned by the GOF.
Note that for the case of only one target, the target
number may be zero.
3. VV is the two-digit viewing number, assigned by GOF, which
tracks the number of scheduled looks at the
particular the viewing number corresponds to:
- different (requested) observations of the same target (e.g., at
different epochs for monitoring),
- different instrument configurations during the same
- different scans for scan-mapping of extended
4. SS is the two-digit sequence number used for identifying
different pointings that make up the same viewing if the SOF decided, for
operational reasons, to split that viewing into more than one chunk or if
it was broken up by e.g. a TOO observation.
5. X, the (optional) 15th character, indicates:
A Slew before observation
Z Slew after observation
S Raster scan observation
R Raster grid observation
0-9 Segments of a long observation (i.e., > 8 hours)
b-r (reserved for) Real-time configuration changes
When not present, it indicates that the data correspond to a regular
pointed observation (< 8 hrs), or the last segment of a longer
2.2.3 Spacecraft Subsystems
Below each ObsID directory are a set of 15 subdirectories, each
containing data files derived from a single spacecraft Subsystem. The
corresponding directory names are as follows:
ace - Attitude Control Electronics & star trackers
acs - Attitude Control System
cal - References to files in the Calibration Database
clock - Time delta correction data from Mission Operations Center
eds - Experiment Data System housekeeping
fds - Flight Data System
gsace - Gimbals and Solar Array Control Electronics
hexte - HEXTE science and housekeeping data
ifog - Interferometric Fibre Optics Gyroscope
ipsdu - Instrument Power Switching and Distribution Unit
orbit - Orbit ephemeris from the Flight Dynamics Facility
pca - PCA science and housekeeping data
pse - Power System Electronics
spsdu - Spacecraft Power Switching and Distribution Unit
stdprod - Standard products generated by the XSDC
Although they represent the lowest rung in the directory hierarchy, the
Subsystem directories do not necessarily contain files of one type. In
most cases, a further division is made based on Application, the term used
for a distinct source of telemetry. In the case of the PCA, the six Event
Analyzers are considered applications.
Also in the ObsID directory are various index files (FI*) used by XDF
(the XTE Data Finder), the XTE filter tools, and other software to
navigate the underlying data files.
2.2.4 Example of Directory Structure
Within W3BROWSE, the relevant database to search for RXTE pointed (PCA
and HEXTE) data in the public domain is XTEPUBLIC. An example will
clarify how XTEPUBLIC relates to the ObsID and manifold directory
structure. Searching XTEPUBLIC for observations of 4U1538-52 yields the
The first few ObsIDs contain data from a series of observations of the
source. The last two ObsIDs listed above contain slew data from before
and after the first observation. (Other ObsIDs will appear from other
observations of the same source; if these are limited to slew files, the
data from the pointed observations are not yet public.)
For the user of anonymous FTP, the archival directory containing PCA
data from the first ObsID (the pointed observation) would be seen as:
Note that although you can work your way down the directory structure
using FTP, it is often more convenient to use W3Browse to find and
retrieve the data, or retrieve a whole ObsID using FTP.
3. Browsing, retrieving and analyzing PCA and HEXTE data
3.1 When will the GO data I'm interested in go public?
The RXTE GOF has developed the Public Data Web tool (http://heasarc.gsfc.nasa.gov/docs/xte/xte_public.html
) to allow users to find out when proprietary data will go public.
Users can search on target name, PI, proposal number, complete ObsID,
and/or date. For example, to find out what data went public in March
1998, one would enter in the appropriate box "Mar-*-1998". Note that if
the data have not yet been (re)processed by XSDC, the go-public date will
not be known and the observations will not appear in the output from the
Public Data Web tool.
The user may also search the XTEPUBLIC catalogue using the HEASARC's
W3BROWSE utility (http://heasarc.gsfc.nasa.gov/W3Browse/) to
find out which datasets are currently available.
3.2 How do I find and retrieve the data I want?
There three methods for retrieving public data: the web-based W3Browse,
anonymous FTP, and the XTE Data Finder (XDF).
3.2.1 Using W3Browse
W3Browse allows you to browse available HEASARC datasets via the Web.
Link to the main W3Browse page, and choose the "Advanced" version (to
allow greater search flexibility). In the "Advanced" interface, check the
RXTE mission box, choose to search either by object name/coordinates or by
parameters, and then begin your search:
a. Choose the XTEPUBLIC catalogue.
b. Enter search criteria into appropriate boxes. If you chose
to search by name, you might enter e.g. 4U1907+09 into the Object
Name or coordinates box, and click Submit. To search by parameters,
you'll click again to begin your search, and perhaps choose data by
proposal number (called PRNB within W3BROWSE), ObsID etc. Then click
c. A successful search will display the corresponding catalogue
entries, i.e. the ObsIDs, one per line. Check the box(es) to the left of
the ObsID(s) you're interested in. (Remember that ObsIDs ending in an A
or a Z are slews. In most cases, the target name should also indicate if
the data correspond to a slew.)
d. Below this listing, you'll find a choice of datasets. You
will probably want one of the FULL datasets, e.g. FULL RAW OBSID or FULL
RAW PROPOSAL. (The REDUCED OBSID DATA contains standard products for each
observation which may be useful as an indication of the strength of the
source and the duration and coverage of the observation, but these are not
intended for use in any detailed scientific analysis.) Choose one by
clicking on it to highlight it.
e. Then click on the radio button to "Retrieve data products in
selected categories for selected observations." Note that the default
setting, "List all data products for selected observations" is not a
helpful option for RXTE datasets, which can contain many hundreds of files
- unless you know you only need one or two.
f. Click on Submit. W3BROWSE will now construct the
appropriate tar command, and tar up the data you have requested. This may
take a while.
g. W3BROWSE will bring up a new page when your tar job is
complete. You may now click on "Download TAR file" to initiate the
transfer of the tar file to your host machine. Make sure you have enough
disk space! - the total size of the tar file will be indicated.
h. When untarred, your data files and index files will appear
in the usual configuration for RXTE datasets, as described above.
3.2.2 Anonymous FTP
Anonymous FTP can be a convenient alternative to W3BROWSE, if you don't
wish to tie up your Web browser for the duration of the data retrieval.
Here's an example of the steps you might follow:
1. "ftp legacy.gsfc.nasa.gov", giving 'anonymous' as your name, and your Email address as the password.
2. "cd xte/data/archive" (stopping to read the information presented here).
3. "cd AO1". "get P10145.tar" to obtain all the public data
from this proposal. The FTP server will automatically create the
tarfile on your home machine in the directory from which you initiated the
FTP run - make sure you have sufficient space. You might want to "cd
P10145" and obtain a subset of the data (e.g. "get 10145-02-02-0.tar").
There is no advantage to pulling over a gzipped file using e.g. "get 10145-02-02-00.tar.gz". All the datafiles within the directory structure are already gzipped, and the savings in transfer time will be minimal or non-existent.
If you try to FTP data which is still proprietary, you'll end up with a tarfile containing clock and orbit information, calibration files and index files, but no science data.
The XTE Data Finder (XDF) is a Graphical User Interface that may be
used to search for data and retrieve them from the HEASARC archive. One
advantage of using XDF is that it allows you complete access to
information about the data modes used for each observation.
1. Obtain the Top-Level Master Index File (FMI), using
W3BROWSE. (To do this, select any random RXTE observation
using steps 1 and 2 of the recipe above in 3.2.1, then choose "XTE
Top-Level FMI" as the category of data product to
2. Once you have this FMI on your home machine, invoke XDF.
3. Change the default FTP setting in the top left hand corner
of the GUI from "FTP No Files" to "FTP Any Files", and enter
the path of the directory where you're keeping the FMI into the "Path:"
4. Click on the "Make ObsList" button. While this button stays
lighter than the surrounding GUI, XDF is hard at work reading the FMI.
5. You can now scroll up and down through the entire available
archive in the "Observations" window, or make selections by source and/or
date in the "Sources" or "Time Ranges" window. (To enter a list of
sources, click on the "Edit" button.)
6. Once you've chosen the sources or data of interest, click on
"Make AppIDConfigList", choose the instruments and data configurations of
interest, and click on "Make Filelist".
During this stage, XDF will automatically connect to the HEASARC
archive and download any necessary index files, or any data files that you
request. The progress of these FTP sessions will be logged in a separate
window. Data will be saved on your current disk in the familiar RXTE
3.3 Cautions and Caveats
RXTE data will be released into the archive on a weekly basis.
Observations newly turning public before 12:01a.m. on a given Saturday
will be available (i) the following Monday, for anonymous FTP access; (ii)
Tuesday, for W3BROWSE access.
RXTE data will go public on an ObsID basis, based on the date when the
tape containing that ObsID was mailed to the original PI. Thus, not all
the parts of a given proposal will necessarily go public at the same time,
particularly in the case of large proposals or long-term monitoring
RXTE datasets may be larger than you are accustomed to, and response
may be slow at peak times, particularly if multiple users are attempting
to get their hands on the same dataset. Please ensure you have sufficient
disk space for the transfers, and try to plan your data retrieval to avoid
times of heavy network traffic. If you have problems, try again
3.4 Analyzing PCA and HEXTE data
A full description of the extensive suite of tools available for RXTE
data analysis is beyond the scope of this article. Complete documentation
for the installation and use of all tools is available online from the GOF
homepage by following the link to "Data Analysis & Processing." Here
are some highlights:
Selective untarring of RXTE data can be performed with the XSUT tool,
which is equipped with a graphical user interface (GUI) and is available
for the first time under FTOOLS 4.1.
Finding your way through the hierarchical directory structure to
identify the data you want to reduce is made easy using XDF. Its
end-product is a list of filenames corresponding to the data you want to
analyze, as chosen by target name, time of observation, instrument, and
Data manipulation and filtering, and the extraction of spectra and
lightcurves, can be performed using the XTE (and other) FTOOLs. You can
use the FLAUNCH GUI to run these tools and keep a log of your
The individual instrument teams play the major role in calibrating
their instruments. The GOF provides a repository for calibration files
and instrument-team-provided software, along with information on their
application. The teams provide documentation on their calibration efforts
on their own Web pages. There are extensive links to this information via
the GOF "About RXTE" and "Data Analysis" pages.
As for all HEASARC X-ray datasets, XSPEC and XRONOS are available to
perform spectral and timing analysis.
Comprehensive instructions for reducing RXTE data are provided in three
complementary online guides:
(i) "The RXTE Getting Started Guide" contains information about
how to read your RXTE data tape, the directory structure on your tape, how
to use XDF to browse your files, and how to install all the necessary
(ii) "The ABC of XTE" provides extensive information about RXTE
data files, data screening and filtering, the extraction of spectra and
light curves, background subtraction, etc.
(iii) "The RXTE Cook Book" provides a large number of
step-by-step recipes for the most commonly-performed RXTE data reduction
tasks. Complete end-to-end recipes are available for the reduction of
PCA, HEXTE and ASM data, along with individual recipes for creating filter
files, working with realtime data, screening and selection of data,
correcting for background and deadtime, creating response matrices,
performing Fourier analysis of PCA data and searching for high-frequency
QPO, performing pulse phase spectroscopy, managing a FITS database,
RXTE data analysts may send queries on any aspect of RXTE to
firstname.lastname@example.org. However, the FAQ page (
http://heasarc.gsfc.nasa.gov/docs/xte/ftools/xtefaq.html) should be
consulted first, to see if it contains the information required.
4. ASM data
4.1 Overview of the ASM Data Products
The ASM Products Database makes available results from the RXTE All Sky
Monitor. These results include light curves for around 300 sources,
intensities in three sub-energy bands (or "colors") for each of these
sources, and a history of the ASM pointings. The database provides these
data files in FITS format. (MIT's ASM Light Curves Overview page provides
GIF images and ASCII files for the light curves).
The database consists of both Quicklook and Definitive versions of the
Products. The Quicklook Products are derived from analysis of the
real-time data performed in the RXTE SOF. They are normally updated every
three hours and include data from up to the previous two weeks. The
Quicklook Products are superceded by updates to the Definitive Products,
which are produced at MIT from the final production data. The Definitive
Products are usually updated weekly.
An important feature of the light curves and color files is that
because two of the ASM Scanning Shadow Cameras (SSCs) have overlapping
fields of view and because data from each camera are analyzed
independently, it is often the case that sources may have multiple
intensity measurements at the same time. These measurements are, however,
independent and may be combined as such.
Quicklook ASM results are also available via the ASM Weather Map. The
map provides an "at a glance" look at the X-ray sky, and an accompanying
table provides the latest x-ray intensity values for over one hundred
active sources in the ASM catalogue. Selecting a source name from the
table produces a plot and ASCII table of the light curve over the previous
4.2 Access to the ASM Data Products
4.2.1 Via FTP
The ASM Data Products are accessible from the ASM Data Products Page.
Using direct anonymous FTP to legacy.gsfc.nasa.gov, the Products may be
found in the following subdirectories of
The Definitive Products:
Light curves: definitive_1dwell/lightcurves
The Quicklook Products:
Light curves: realtime_current/lightcurves
4.2.2 Via W3Browse
For the Definitive Products:
Choose the "XTE All-Sky Monitor Long-term Observed Sources"
(XTEASMLONG) catalog. After giving the name or coordinates of a
particular source, the user may then either choose to retrieve the
products individually or in data product sets. The STANDARD set consists
of both the light curve and the color file for the object. The POINTINGS
set consists of all of the weekly ASM pointing files.
For the Quicklook Products:
Choose the "XTE All-Sky Monitor Quicklook Observed Targets"
(XTEASMQUICK) catalog. After giving the name or coordinates of a
particular source, W3Browse will list the intensity values at times
spanning the previous two weeks. After choosing any of these rows, the
user may then choose to retrieve the quicklook products either
individually or in the data product sets. As with the definitive
products, the STANDARD set consists of both the light curve and the color
file for the object. The POINTINGS set consists of all the daily ASM
pointing files presently in the real-time archive.
4.3 Analysis of ASM Products
The ASM Data Products can be analyzed using the FTOOLS software. See
the "Working with the ASM Data Products" recipe in the RXTE Cook Book for
The construction, population and maintenance of the RXTE Archive is a
collaborative effort between several groups at NASA/GSFC. The XTE Science
Data Center (XSDC) perform the pipeline processing to convert the
production data into FITS format and distribute data to GOs. They provide
these data to the GOF, along with the distribution logs stating when each
dataset should go public. The XSDC pipeline is based around the core
dmIngest and XFF (XTE Fits Formatter) software provided by the SOF and GOF
respectively. The GOF is responsible for data transfer and population of
the Archive and some quality control functions. HEASARC handles the
database tables for the W3BROWSE online interface, and maintains the
Archive. While a rather large cast of characters contributed to this
scheme, we name here only the principal architects: Arnold Rots, Randy
Barnette, Bob Patterer, Mike Tripicco, and Ed Sabol.
The RXTE GOF homepage: http://rxte.gsfc.nasa.gov/
The HEASARC homepage: http://heasarc.gsfc.nasa.gov/
The Technical Appendix to the NASA Research Announcements; can be obtained from
The ASM Data Products page:
The PCA Team homepage:
The HEXTE Team homepage:
The ASM Team homepage: http://space.mit.edu/XTE/XTE.html
(All of the above are accessible via the GOF pages.)
HEASARC Home |
Last modified: Monday, 19-Jun-2006 11:40:52 EDT