NAME

mxpipeline

USAGE

mxpipeline ra dec starttime stoptime

DESCRIPTION

mxpipeline is an end-to-end processing tool to take raw MAXI data and generate images, lightcurves, spectra, response files, and event files. It requires that MAXI event and auxiliary files have been downloaded, the FTOOLS environment defined, and that CALDB with MAXI files is available. Data may be processed from GSC and/or SSC instruments, and GSC may be from one or the other of GSC_LOW or GSC_MED.

The tool runs a series of MAXI-specific and general FTOOLS to select raw event tiles within a radius of the given source position, calculate good time intervals, generate standard products, and perform all processing needed to make those products. Lightcurves are generated with scan-resolution (one value per ISS orbit) and bin-resolution time intervals, based on the setting 'lcbin'.

Although there are a very large number of possible input parameters, most have default settings appropriate for common scenarios. The configurations of most interest are likely:

Most other customized settings (selecting specific voltages, using local files in the place of reference data or the MAXI calibration database, etc.) are useful for only very specific analysis goals. Alternatives to CALDB should be used with caution.

PARAMETERS

ra [real] (J2000; degrees)
Right ascension of the source.

dec [real] (J2000; degrees)
Declination of the source.

tstart [string]
Start date of the observation. The parameter can be entered in the format yyyy-mm-ddThh:mm:ss or yyyy-mm-dd (to start at 00:00:00).

tstop [string]
Stop date of the observation. The parameter can be entered in the format yyyy-mm-ddThh:mm:ss or yyyy-mm-dd. If hh:mm:ss is not included, the stop date will include all data up to and including the end of the stop date (i.e. to end at 23:59:59).

(eband_fname = "NONE") [string]
Up to three different energy bands can be supported. If set to "NONE" (default) or "REFDATA", the default energy band specification file provided in REFDATA will be used. This defines 2-6 keV and 6-20 keV bands for GSC, and a single 0.7-7.0 keV band for SSC. If a file name is given, that file needs the following format:
# GSC 
# conversion 50 eV/ch 
#  min_ch    max_ch  min_ene   max_ene 
   40        120     2.0       6.0
   120       400     6.0       20.0
# SSC 
# conversion 3.65 eV/ch
#  min_ch    max_ch  min_ene   max_ene
   192       1918    0.7       7.0 
Lines that begin with # indicate comments, but the lines "# GSC" and "# SSC" are required. The energy range settings are used to make images and lightcurves: spectra will reflect the full range supported by the MAXI instrument independent of these settings.

(object = "target") [string]
The name of the target object: this will be used for file names and setting the OBJECT keyword value in FITS format product files. Default is "target". Note that in addition to be set as the OBJECT keyword, this value is also used to set the rootname of all files, but transposed to make valid unix filenames without special characters. E.g. 'object="Cyg X-3"' would set OBJECT in all files to be "Cyg X-3" and all files will start with 'cygx3'.

(detector = "gsc_low") [string])
The name of the MAXI detector(s). Should be one of 'gsc_low', 'gsc_med', or 'ssc' ('ssc_med' is also accepted). The default is 'gsc_low'. To process both GSC and SSC data, specify both separated by a comma (e.g. 'gsc_med,ssc_med'). Only one GSC datamode can be supported at a time (i.e. 'gsc_low,gsc_med' is not supported).

(datapath = "./obs") [string]
Path to the raw data. By default, this is './obs', matching the output settings in mxdownload_wget.pl, the recommended tool for downloading MAXI raw data. The event tiles are arranged in the standard way ([datapath]/MJD##000/MJD####/events/[detector]/...) below this root directory.

(auxpath = "trend") [string]
The local path to the list files for the assorted auxiliary files. The default is "trend". If the directory does not yet exist, it will be created. If existing files are found in this location, they are checked to ensure they include the requested date range and replaced if not. Replacement is blocked by setting 'clobber' to 'no' (see below).

(outpath = "products") [string]
The local path in which to place final product files. The default is "products". If the directory does not yet exist, it will be created.

(leapsecfile = "REFDATA") [string]
If set to "REFDATA" (default), use the built-in reference file for defining leap seconds. If set to "CALDB", use the leap second file in the MAXI CALDB. Otherwise, use the specified file name.

(gsc_exclude = "3,6,9,10,11") [string]
Provide a comma-separated list of any GSC camera(s) to exclude from consideration. Default is "3,6,9,10,11" to exclude cameras with known issues through most of the mission timeline. Set to "NONE" or ' ' to include all possible cameras. Note that some cameras will be excluded if simulated background is requested.

(attlist = "NONE") [string]
Name for the output attitude list file created by combining the .att attitude files in the auxiliary data. The use of .att files is supported, but they are not considered as reliable as the .iat version of the attitude files. This parameter is set to "NONE" by default.

(isalist = "isalist.fits") [string]
Name of the output file cataloging the complete set of auxiliary .isa files for all dates they are available. These files contain the solar panel ancillary joint angles used in combination with the solar paddle angles (.isp files; see below) to determine scan intervals in which the field of view is obscured by the ISS solar panels. The default is "isalist.fits". This is ignored in SSC processing as the solar panels never obstruct those cameras.

(isplist = "isplist.fits") [string]
Name of the output file cataloging the complete of auxiliary .isp files for all dates they are available. These files contain the solar paddle angles and when used in combination with the .isa files (see above), determine times when the ISS solar panels obstruct the GSC field of view. The default setting is "isplist.fits". This is ignored when processing SSC data as the solar panels do not obstruct the SSC at any time.

(iatlist = "iatlist.fits") [string]
Name for the output attitude list file created by combining the .iat attitude files in the auxiliary data. The default is "iatlist.fits". The .iat versions of ISS attitude is recommended for MAXI data analysis.

(iaclist = "NONE") [string]
Name of the output attitude file created by listing the .iat files for all dates they are available, and .att on days where .iat is not available. This format is obsolete as reprocessing of MAXI data provides .iat for all valid MAXI dates. The default setting of "NONE" disables the generation of this file.

(vclist = "vclist.fits") [string]
Name of the output list file of GSC housekeeping files used to monitor "veto carbon" conditions. The default is "vclist.fits", but can be set to "NONE" unless GSC processing with simulated background events will be performed as this is the only application that needs these files.

(search_radius = 8.0) [real] (degrees)
Search radius around the target sky location. The default value is 8.0 degrees. All raw data tiles containing data within this search radius will be collected, and image files will contain events within this radius.

(nx = 200) [integer] (pixels)
The number of pixels on the X-axis when mapping events to X positions in the image array. The default value is 200 pixels. If set to a negative number, the code will read the value set for NX in the raw event MAXI tiles or 2 x 'search_radius' / 'pixsize', whichever is larger.

(ny = 200) [integer] (pixels)
The number of pixels on the Y-axis when mapping events to Y positions in the image array. The default value is 200 pixels. If set to a negative number, the code will read the value set for NY in the input raw event MAXI tiles or 2 x 'search_radius' / 'pixsize', whichever is larger.

(pixsize = 0.1) [real] (degrees)
The size of pixels, in degrees, in image arrays. If set to a negative number, the code will read the value for pixel size from the input raw event MAXI tiles. The default value is 0.1 degrees

(src_regfile = ' ') [string] (filename)
If set to "NONE" or blank (default), the code will generate a new circular source region file around the source defined by 'src_radius' (see below). If set to a filename, use that file to define the source region and do not generate any new region file.

(bkg_regfile = ' ') [string] (filename)
If set to "NONE" or blank (default), the code will generate a new background region file defined by input parameters 'bkg_ra', 'bkg_dec', 'bkg_outer_radius' and 'bkg_inner_radius' (see below) unless simulated background data is requested (in which case the source and background region files are identical). This setting is ignored if 'bkg_outer_radius' is set to a negative number (i.e. no background will be defined.) If set to a filename, use that file to define the background region and do not generate any new region file.

(src_radius = 1.6) [real] (degrees)
The radius of a circle around the source used to define the source region. The default setting is 1.6 degrees.

(bkg_ra = -1.0) [real] (J2000; degrees)
Right ascension of the center of the background region. Set to a negative value (default) to use the same central sky location for source and background. Valid positive values are 0 - 360 degrees in J2000 coordinates.

(bkg_dec = -1.0) [real] (J2000; degrees)
Declination of the center of the background region. Ignored if the background is centered on the source (i.e. 'bkg_ra' is less than zero) or if no background is requested (i.e. 'bkg_outer_radius' is less than zero). Valid values are -90 - 90 degrees in J2000 coordinates.

(bkg_outer_radius = 3.0) [real] (degrees)
Outer radius of the background region. Set to a negative value to skip creating a background region file. Default is 3.0 degrees.

(bkg_inner_radius = 1.7) [real] (degrees)
Inner radius of the background region, excluded from the background to define an annulus. If the background region is centered on the source, this should be at least 0.1 degree larger than source radius. Set to 0 to skip (i.e. background region is a circle, not an annulus). Default is 1.7 degrees.

(catalog = "REFDATA") [filename]
A bright source catalog in FITS format. Set to "NONE" to skip. Set to "REFDATA" (default) to use the default MAXI bright source catalog file included with HEASoft. Set to a filename to use a different catalog of bright sources.

(confusion_radius = 1.0) [real] (degrees)
Radius around any bright sources in the catalog(s) to exclude. Ignored if no catalogs are set. Default is 1.0 degrees.

(reject_radius = 0.7) [real] (degrees)
Radius around input source location to consider as coincident (i.e. if a catalog source falls within this radius, it is considered to be the input source and therefore NOT excluded). This prevents problems caused by minor discrepancies for source locations between the catalog(s) and user input source coordinates. Default is 0.7 degrees.

(usercat = "NONE") [filename]
Set to "NONE" to ignore (default). A user supplied catalog of sources. This should be a text file with comma separated values, one source per line, in either of the following formats: RA, DEC; or Name, RA, DEC. Sky coordinates must be in J2000 expressed as decimal degrees. This file can supplement the catalog file or replace it: in the latter case, by setting the bright source catalog to "NONE".

(binwidth = 1) [real] (s)
The size of time bins to use in field-of-view GTI files. The default setting is 1 second. Note this is not the same as 'dt', the time bin size used for computing effective area for spectra (see below).

(threshold = 1.0) [real] (count/s)
The minimum rate above which data will be considered when looking at event rates to construct GTI. The default setting of 1.0 will include in GTI consideration time bins with rates greater than 1.0.

(timegap = 50.0) [real] (s)
The minimum interval of time between adjacent events to be considered part of the same scan, in seconds. The default is 50.0 seconds.

(usergti = "NONE") [filename]
Limit the consideration of events to only intervals defined by this file. If set to "NONE" (default), no user GTI file will be used.

(maxlines = 5000000) [integer]
In longer time intervals, event files or light curves can be generated with are large enough to cause memory issues. Breaking processing into pieces may be helpful to avoid processing issues by setting an upper limit on the number of data rows (events or light curve bins) to load at once. The default setting is 5000000.

(hvlist = "CALDB") [string]
A list of high voltage state settings for GSC. This may be a comma-delimited list, or the name of a text file containing a list of files, one per line, preceded by an '@' character. If set to "CALDB" (default), the high voltage history is obtained by reading the MAXI calibration database. Ignored if the detector is SSC.

(ssdockinfo = "CALDB") [filename]
Name of the file containing information about the presence of a docked space shuttle which obscures the GSC instrument field of view. Set to "NONE" to disable. The default setting of "CALDB" will obtain this information from the MAXI calibration database. Not applicable to SSC processing as the space shuttle does not obscure these cameras when docked.

(teldef_gsc = "CALDB") [file list]
A list of files containing the TELDEF information for the GSC cameras. This may be a comma-delimited list, or the name of a text file containing a list of files, one per line, preceded by an '@' character. The default setting of "CALDB" will obtain this information from the MAXI calibration database.

(teldef_ssc = "CALDB") [file list]
A list of files containing the TELDEF information for the SSC cameras. This may be a comma-delimited list, or the name of a text file containing a list of files, one per line, preceded by an '@' character. The default setting of "CALDB" will obtain this information from the MAXI calibration database.

(colea_gsc = "CALDB") [file list]
A list of files containing the collimator effective area maps for each GSC camera. This may be a comma-delimited list, or the name of a text file containing a list of files, one per line, preceded by an '@' character. The default setting of "CALDB" will obtain this information from the MAXI calibration database.

(colea_ssc = "CALDB") [file list]
A list of files containing the collimator effective area maps for each SSC camera This may be a comma-delimited list, or the name of a text file containing a list of files, one per line, preceded by an '@' character. The default setting of "CALDB" will obtain this information from the MAXI calibration database.

(angsepar = 0.3) [real; degrees]
If using observed background, the amount of tolerated offset between the center of the source and background pointing before the code presumes background and source sky positions are not concentric.

(max_colphi = 37.6) [real] (degrees)
The maximum collimator phi angle considered valid: values outside this range are treated as out of view. The default is 37.6 degrees, appropriate for GSC. Should be set to 45.0 for SSC which has a wider field of view.

(min_colea = 1.25) [real] (cm^s s)
The minimum collimator effective area in cm^2 s (default GSC=1.25, SSC=0.25). Intervals with less than this effective area will be flagged as "low exposure".

(min_obscured_spss = 1.5) [real] (degrees)
The minimum angular separation between the instrument pointing direction and the solar panels and/or spacecraft: if an interval includes time less than this angle from either of these obstruction, an obstruction flag will be set. Ignored when processing SSC data.

(min_obscured_wire = 0.75) [real] (degrees)
The minimum separation from central wire which obscured the view of the GSC cameras. Ignored when processing SSC data.

(angsepar = 0.3) [real] (degrees)
If using observed background, the amount of tolerated offset between the center of the source and background pointing before the code presumes background and source sky positions are not concentric.

(vlfilt = yes) [boolean]
A flag to indicate whether to filter out incomplete scans with very low total area ("yes" to exclude (default); or "no" to retain them).

(crabratefile = "CALDB") [filename]
Name of the Crab rate map file, which gives the observed Crab spectra at different locations of the detector and used to compute a modelled spectra. If set to "CALDB" (default), the file is obtained from the MAXI calibration database.

(bexsigmafile = "CALDB") [filename]
Name of the file containing the position-dependent one-dimensional point spread function (the X direction at the Beryllium window). If set to "CALDB" (default), the file is obtained from the MAXI calibration database.

(backmodel = no) [boolean]
Setting to generate GSC simulated background evets. Default is 'no'. If set to 'yes', simulate events over the source region to model background accounting for contributions from the background noise of the instrument (NXB: Non X-ray background), additional background X-ray noise from a Soyuz spacecraft if present, and cosmic X-ray background (CXB) contributions.
NOTE: The background model is not supported for SSC, nor for certain specific GSC cameras and voltages (6, 9, 10 & 11 for both 1550V and 1650V, plus 0, 3, 4, & 5 for 1550V). The setting is therefore ignored for SSC, and unsupported camera/voltage combinations will not be included in lightcurves and spectra products for GSC.

(soyuzgti = "CALDB") [file list]
A file defining the times when a Soyuz spacecraft is docked at the ISS. Soyuz carries an radioactive source, which increases the MAXI background noise rate. The default setting of "CALDB" will obtain this information from the MAXI calibration database.

(bgtemplatefile = "CALDB") [filename]
Name of the background template file containing the camera and voltage specific settings for the cosmic X-ray background, non-X-ray background, and Soyuz-generated background used to simulate background events. If set to "CALDB" (default), the file is obtained from the MAXI calibration database.

(sim_enelo = 2.0) [real] (keV)
Lower energy bound for simulated background events. Ignored if not generating simulated background. The default is 2.0 keV and the maximum range supported is 0.0 - 25.0 keV. Note, however, that CALDB only generates meaningful values in the 2.0 - 20.0 keV range.

(sim_enehi = 20.0) [real] (keV)
Upper energy bound for simulated background events. The default is 20.0 keV and the maximum range supported is 0.0 - 25.0 keV. Note, however, that CALDB only generates meaningful values in the 2.0 - 20.0 keV range.

(sim_scale = 10) [integer]
Simulation scale factor, such that the simulated events are made at 'sim_scale' times the simulated observation data rate. A higher value provides better statistical accuracy, but takes longer to compute. The default value is 10.

(hvsel = 0) [integer]
Flag setting for indicating which high voltage settings to include in data. Allowed settings are :
0 (include both 1650 V and 1550 V data) (default);
1 (only include 1650 V data);
2 (only include 1550 V data). Ignored when processing SSC data.

(lowexpo_limit = 20.0) [real] (cm^2 s)
Threshold value to set a low exposure flag for any individual camera scan. Default value is 20.0 cm^2 s.

(use_c1c2 = yes) [boolean]
A flag, set to "yes" (default) or "no", to include anodes C1 and C2 in the analysis of GSC data. Ignored when processing SSC data.

(lcbin = 1.0) [real] (days)
The size of the bins to use when making binned lightcurve data. Default setting is 1.0 days.

(coltha_max_gsc = 5.0) [real] (degrees)
The maximum absolute value of source theta relative to a GSC camera in degrees. Default setting is 5.0 degrees.

(coltha_max_ssc = 5.0) [real] (degrees)
The maximum absolute value of source theta relative to an SSC camera in degrees. Default setting is 5.0 degrees.

(obscol_dname = "REFDATA") [string]
The directory where the field of view parameter files exist. If set to "REFDATA" (default), these files are obtained from the built-in reference data directory.

(dt = 1) [real] (seconds)
The length of each time step in seconds when calculating theta, phi, and instantaneous collimator effective area used for spectral response function math (NOT the same as the time bins used for field-of-view GTI ('timebin')). Smaller time settings provide better granularity and more precise measurement of total effective area in spectral products at the expense of longer computation time. The default step size is set to 1.0 seconds.

(colssc = "CALDB") [filename]
List of the collimator slat-plane position files for the SSC instrument. This may be "CALDB" (default), a comma-delimited list, or the name of a text file containing a list of files, one per line, preceded by an '@' character. If set to "CALDB", obtain the SSC collimator slate-plate position files from the MAXI calibration database.

(arfcorr_gsc = "CALDB") [filename]
List of ancillary response function (ARF) files for the GSC cameras. This may be "CALDB" (default), a comma-delimited list, or the name of a text file containing a list of files, one per line, preceded by an '@' character. If set to "CALDB", obtain the ARF files from the MAXI calibration data files.

(arfcorr_ssc = "CALDB") [filename]
List of ancillary response function (ARF) files for the SSC cameras. This may be "CALDB" (default), a comma-delimited list, or the name of a text file containing a list of files, one per line, preceded by an '@' character. If set to "CALDB", obtain the ARF files from the MAXI calibration data files.

(rmffile_gsc = "CALDB") [filename]
List of response matrix files (RMFs) for the GSC cameras. This may be "CALDB" (default), a comma-delimited list, or the name of a text file containing a list of files, one per line, preceded by an '@' character. If set to "CALDB", obtain the RMFs from the MAXI calibration database.

(rmffile_ssc = "CALDB") [filename]
List of response matrix files (RMFs) for the SSC cameras. This may be "CALDB" (default), a comma-delimited list, or the name of a text file containing a list of files, one per line, preceded by an '@' character. If set to "CALDB", obtain the RMFs from the MAXI calibration database.

(quanteff = "CALDB") [filename]
List of the quantum efficiency files for the SSC camera. If set to "CALDB" (default), these files are obtained from the MAXI calibration database.

(cleanup = no) [boolean]
Determines whether to delete temporary files. When set to no (default), intermediate files will be retained.

(clobber = no) [boolean]
Overwrites the existing output file if set to yes. Default is no.

(chatter = 1) [integer]
Sets the amount of output from the code. Values range from 0 (little to no output) to 3 (verbose). Default value of 1 provides basic details.

(logfile = "!DEFAULT") [string]
Record output from running this tool in a logfile. If set to "DEFAULT", the output logfile will be named 'mxpipeline.log'. If preceded by the '!', any existing file with the same name will be overwritten; otherwise, this output will be appended to any existing file. No log file made if set to NONE. The default setting is "!DEFAULT".

(history = yes) [boolean]
Records tool parameters in HISTORY keywords if set to yes (default).

(debug = no) [boolean]
If set to yes, the code will provide detailed logging of variables and settings. This can be useful when debugging features in the code or deciphering unexpected error conditions. Default is no.

(mode=ql) [string ql|hl|q]
Mode to query the parameter file. The default value, 'ql’ sets non-hidden parameters to query and learn/remember, which can be useful when repeating similar processes multiple times (such as generating products for multiple sources in the same fashion for each). Acceptable values include: 'ql' (query and learn/remember), 'hl' (hidden and learn/remember), 'q' (query but don’t remember), 'h' (hidden).(Optional)

EXAMPLES

  1. Create products for dates 2010-01-04 to 2010-01-07 (inclusive) using using GSC_LOW data with a source region defined by a 2.2 degree radius circle around the Crab, and obtain background measurements using an annulus around the source with an outer radius of 3.0 degrees and inner radius of 2.3 degrees. Make binned lightcurves with 0.25 day bin size. Overwrite any pre-existing files if found. Include the settings used to make the files in the HISTORY keywords of these product files. The tool will generate a log file named 'mxpipeline.log', clobbering any prior log file with that name. Generates event, spectra, image, and lightcurves, plus a number of additional intermediate products: retain all these files and place them in the directory "products" (default 'datapath').

    mxpipeline ra=83.633083 dec=22.0145 tstart=2010-01-04 tstop=2010-01-07 object="Crab" detector=gsc_low src_radius=2.2 \
      bkg_outer_radius=3.0 bkg_inner_radius=2.3 lcbin=0.25 clobber=yes cleanup=no history=yes
    
  2. Create images, lightcurve, spectra and event files for Cyg X-3 for all days in July 2021. Define the source region to be a 1.6 degree radius circle around the source, and generate simulated background event files for the GSC medium download rate data. Use a local customized energy band definition to get three bands defined for 2-6, 6-12, and 12-20 keV (respectively referred to here as soft, medium, and hard). When making lightcurves, make one day long time bins. Filter out any very low effective area scans when making spectra and lightcurves (However, all data for all available cameras and voltages will be included in the image products.). All file names will have the prefix 'cygx3', while the OBJECT keyword in products to be 'Cyg X-3'. The X and Y columns in the event files and image array in the images will use the same settings as those found in the source raw event tiles (255 x 255 with a pixel size of 0.1 deg/pixel).

    mxpipeline ra=308.107417 dec=40.957750 tstart=2021-07-01 tstop=2021-07-31 eband_fname=cygx3_ebands.txt \
      object="Cyg X-3" detector=gsc_med search_radius=8 src_radius=1.6 backmodel=yes nx=-99 pixsize=-99 \
      lcbin=1.0 chatter=3 debug=yes clobber=yes cleanup=no vlfilt=yes
    
    The energy band file cygx3_ebands.txt would read as:
    # GSC 
    # conversion 50 eV/ch 
    #  min_ch    max_ch  min_ene   max_ene 
       40        120     2.0       6.0
      120        240     6.0       12.0
      240        400     12.0      20.0
    

    The lightcurve file generated, cygx3_gsc_bin.lc, contains six extensions: one for each of the three energy bands, plus one for each of (in order) the ratios hard/soft, hard/medium, and medium/soft. There will be a single image file 'cygx3_gsc.img' with a primary image array and two image extensions which contain, respectively, the soft, medium, and hard energy band images.

    In this particular example, events are found in the MAXI event tiles from GSC Camera 1 & 6, but are then excluded from consideration because the simulated background generator is not enabled for these cameras given their voltages states during this time interval. The typical observer will have no a priori knowledge of camera availability or voltage settings: the code automatically handles these conditions and provides warnings when cameras need to be excluded for unsupported voltage conditions.

    SEE ALSO

    mxproduct, mxdownload_wget.pl

    LAST MODIFIED

    September 2025 (mxpipeline v0.939b)