These are release notes for FTOOLS v4.1, released 7 May, 1998. In this document we discuss the new tools and scripts in version 4.1 and highlight the major changes since version 4.0. For each of the FTOOLS subpackages below we list the tools which are new in v4.1 and briefly describe the major changes to preexisting tools. For more information on any FTOOL type "fhelp" once you've installed the package. "fhelp ftools" will provide a full listing of all the tools available in v4.1, while "fhelp " lists all of the tools in that package. ===== MISCELLANEOUS CHANGES ===== The refdata file scc_to_utc.tfits has been updated and is valid through October 19, 1997. ===== CFITSIO/FITSIO ===== One major change in this release, which should be largely transparent to users, is that the FITSIO library routines which are called by the Fortran FTOOLS tasks to read and write FITS files, have been replaced by CFITSIO routines, written in C. CFITSIO now contains a set of Fortran-callable routines with exactly the same calling sequence as the old FITSIO subroutines, so any software that previously used FITSIO can be relinked against the CFITSIO library without any change. The advantages of using CFITSIO are that it generally produces faster code, and it enables Fortran programs to access the new features in CFITSIO (see below) which were previously only available to C programs. As a result of this change, the FITSIO library is no longer routinely built in FTOOLS, and instead the FITSIO library is now just a link to the CFITSIO library. The FITSIO source code is still included in this release, however, just in case Fortran programmers discover an incompatibility with the CFITSIO library. One of the new features in CFITSIO is that it can open and read input FITS files that have been compressed with the gzip (.gz file extension) or with the Unix compress (.Z extensions) programs. This only works however if the file is opened with readonly access, and not in cases where the FTOOLS task writes to or modifies the input file. Specifying the .gz or .Z extension is optional, so if the FTOOLS task cannot open the specified input file, it will automatically try to open the file with the same root name and with the .gz or .Z extension. Another new feature in CFITSIO which affects all the FTOOLS tasks is the ability to 'pipe' FITS files between tasks using the stdin and stdout file streams. A '-' (the dash sign) should be specified as the name of the input or output FITS file to signify that the file should be read from the stdin stream or written to the stdout stream, respectively. Users may create a pipeline of FTOOLS tasks which pipe the output from one task to the input of the next task. As a simple example one can type the following command: fmemsort myfile.fits.gz - time heap | fdump - This will run the fmemsort task, sorting the compressed FITS file 'myfile.fits.gz' on the 'time' column in the first extension using the heap sorting method. The output file of fmemsort is '-' so the FITS file is written to stdout, which gets piped to the fdump task (using the Unix vertical bar "|" piping command). Connecting tasks together in this way will generally improve the speed of the pipeline because it eliminates having to write then read back intermediate FITS files on magnetic disk. Both of these new CFITSIO features depend on the system having enough physical memory or swap space to be able to make a temporary copy of the FITS file in memory, so it may not be possible to read large compressed FITS files, or pipe large files on machines with small amounts of memory. If that is the case, the FTOOLS tasks may still be used in the conventional way by reading and writing (uncompressed) FITS files on disk. ===== ASCA ===== (*) For complete details and up-to-date information on using FTOOLS v4.1 to analyze ASCA data, please refer to: http://heasarc.gsfc.nasa.gov/docs/asca/ahp_proc_analysis.html ===> New since v4.0 <=== mkfilter2 -- New, an ftoolized version of the standalone program that creates xselect filter files. Note that some errors that were silently ignored in the standalone version are now reported; in particular, warnings are issued for faQparam errors, such as bins outside the limits of the attitude file. The final results agree with the standalone version, however. ===> Major changes since v4.0 <=== addascaspec -- New option "-q" displays commands without executing them. Updates the BACKSCAL keyword. ascaeffmap -- Should now get the correct RMF from the CALDB in the case of SIS. ascascreen -- This includes a more sophisticated screening criterion for the Sn_PIXLm monitor counts for the SIS. This should generally increase the exposure time of the screened event files for recent 2- and 4-CCD mode data as well as for 1-CCD mode data with a bright source. sisrmg -- Can now obtain its calibration files from the Calibration Database (CALDB). To do this, specify "CALDB" (must be in capitals) for the ecdata, phtopi, echosh, and/or rddhis parameters. The default is to read them from the ftools/refdata/ directory as before. Remember to source the script caldbinit.{c}sh first. IMPORTANT: sisrmg when compiled WITH optimization on Linux systems (ONLY, as far as we know), gives incorrect results! The Makefile now explicitly turns off optimization when compiling on Linux. ===> Bug Fixes and Minor changes since v4.0 <=== ascaarf -- Fixed bug in convolving with the GIS PSF - only mattered at extreme off-axis positions. Performance note: when compiled with gcc optimization enabled on Linux, we have had reports that this runs several times *slower* than if it is compiled without gcc optimization, but with g77 optimization. Because of the large number of variables in determining the exact cause of this problem, we have left the gcc optimization ON by default. If you have unsatisfactory performance, we suggest you try re-compiling without gcc optimization, and please let us at ftoolshelp know what you find, so that we can perhaps track down the cause. correctrdd -- Fixed a number of bugs. faintdfe -- Performance will often be improved, due to relaxation of a tolerance in the Romberg integration routine. ===== CALTOOLS ===== ===> New since v4.0 <=== c_caldbflag -- Switches flag status of calibration files in a CIF (C version of caldbflag). c_caldbinfo -- Checks whether a local CALDB is correctly installed (C version of caldbinfo). c_crtcif -- Create an empty Calibration Index File (CIF) (C version of crcif). cifcadd -- Add a new column, CAL_ORIGIN, to a CIF. ===> Major changes since v4.0 <=== addrmf -- HISTORY keywords from first input RSP extension are preserved. caldbflag -- Accounts for extra CAL_ORIGIN columns (if present). New option (parameter) to switch DETNAM flag. gcorpha -- Modified to behave more like gcorrmf. marfrmf -- HISTORY keywords are now preserved in output file. Now uses NAXIS2 from ARF file to set size of the ARF dynamic arrays. ===> Bug Fixes and Minor changes since v4.0 <=== genrsp -- Fixed a bug which caused a crash if response had more than 32767 channels. Minor bug fixes. mudcif -- Accounts for extra CAL_ORIGIN columns (if present). Default value of qualswitch parameter changed to "yes". rbnrpsf -- A bug has been fixed which caused RBNRPSF to set the wrong area-weighting before running hrirpsf/pcrpsf. udcif -- Minor bug fixes. ===== EINSTEIN ===== ===> No changes since v4.0 <=== ===== EXAMPLES ===== ===> New since v4.0 <=== (NONE) ===> Major changes since v4.0 <=== cdummyftool -- Changed to use new libcftools.a routines which eliminate the need for C ftools to use cfortran.h directly. This is significant in that the standard practice for C Ftools has changed. ===== FIMAGE ===== (NOTE: F2DHISTO actually was in v4.0 but was not documented in the Release Notes or the user's guide.) ===> New since v4.0 <=== chimgtyp -- Converts (changes) the data type (BITPIX) of an image array of the FITS image extension. f2dhisto -- Compute a 2-dimensional histogram from 2 input columns which give the x and y coordinates of a series of events. fimgextract -- Extract a (vector) column from a table or a subset from an array, optionally summing over the rows. fimhisto -- Creates a histogram of the image extension. flst2im -- Create an output FITS image file from input list file containing a list of the coordinates and respective values. fmedian -- Convolve a FITS image with a median value within a user-defined rectangular box. ===> Major changes since v4.0 <=== farith * MIN/MAX operators added * Handling of input images/vectors generalized to allow operands of differing dimension. The tool "expands" the smaller operand to the dimensions of the larger. * Vectors can be specified on the command line for either operand via a comma-separated list. New parameters determine the dimensions of the vector. * The preceding two features allow operations between an image file and a constant entered on the command line, making the tool FCARITH unnecessary. ===> Bug Fixes and Minor changes since v4.0 <=== "clobber" parameter added to all relevant fimage tools, allowing a newly-created FITS file to overwrite a pre-existing file. fimgmerge -- Bug on Alphas fixed; system-independent image-handling bug fixed. fimgstat -- Bug on Alphas fixed. fimgtrim -- Bug on Alphas fixed. fmaskfilt -- Fixed bug that caused the program to fail if the file contained WCS keywords. fvec2img -- Potential divide by zero error prevented. ===== FUTILS ===== ===> New since v4.0 <=== (NONE) ===> Major changes since v4.0 <=== fcalc * Calculations with vector columns supported * Unlimited-length expressions supported * inull and anull parameters added for setting the NULL value in the output column fcreate * Creation of vector columns and complex number columns supported * Can optionally read the FITS data from stdin, allowing Unix piping fimgcreate * Can optionally read the FITS data from stdin, allowing Unix piping fmemsort * Multiple column sorting supported * Can optionally purge rows with duplicate sort keys * (Bug Fix) Gracefully handles tables with zero rows fplot * Supports unlimited number of data points (Plotter ignores error bar data if more than 2000 points, though.) * If binmode is SUM or MEAN, vector columns can be plotted against scalar columns or other binned vector columns of different dimensions. * When plotting indexed data (ie, no X-column) over multiple graphs (ie, number of points greater than the maxpts parameter), the index value will be continuous over all the graphs. fselect * Calculations with vector columns supported * Unlimited-length expressions supported * % done progess report sent to screen while running fsort * Can optionally purge rows with duplicate sort keys * (Bug Fix) Comparison of complex numbers handled correctly ===> Bug Fixes and Minor changes since v4.0 <=== fdump -- Uninitialized variable now explicitly initialized to zero; blank line at end of output removed. finterp -- INDEF handling fixed. flookup -- Updates the NEVENTS keyword when using an ASCA events file. fmerge -- Failure to transfer data in bit columns now fixed. fverify -- Tolerate variable-length columns with 0 elements in some rows. ===== GRO ===== There are new FTOOLS for the analysis of BATSE occultation data (NHIS type). The FTOOL BODGETVP reveals which CGRO viewing periods are contained in the data, and the start and stop Truncated Julian Dates of these viewing periods. The FTOOL BOD2PHA converts the occultation data to PHA files in XSPEC format. A third FTOOL to make RMF files in XSPEC format is under development, and will be available in the next release. In the mean time, a VMS version can be obtained from Chris Shrader (shrader@grossc.gsfc.nasa.gov). The new EADDMAP ftool adds or subtracts 2 EGRET counts maps as well as the corresponding exposure and intensity maps. If requested by the user, only the counts maps may be generated. The FTOOL can add maps of different sky regions, bin sizes, and coordinate system (celestial or galactic). The program can combine observations of the same region for different viewing periods, or produce maps of expanded regions such as galactic disk or all sky maps. ===> New since v4.0 <=== bod2pha -- Makes XSPEC PHA file from BATSE occultation data. bodgetvp -- Gets viewing period info from BATSE occultation data. eaddmap -- Add or subtract two EGRET maps. ===> Bug Fixes and Minor changes since v4.0 <=== fbdrm2rmf -- Minor bug fixes. fesdb2rdf -- Now inserts a comment instead of a blank line in the header. ===== HEASARC/EXTRACTOR ===== ===> New since v4.0 <=== (NONE) Note: the extractor was formally moved into the heasarc package. ===> Major changes since v4.0 <=== hkexpand -- Support was added for single precision, short integer, byte and character columns. hkunexpand -- Support was added for single precision, short integer, byte and character columns. ===> Bug Fixes and Minor changes since v4.0 <=== addspec -- Changed to work with a change made to addrmf, and bug fixed which caused certain legal filenames not to be recognized correctly. chkarf -- Minor bug fixes. chkrmf -- Minor bug fixes. extrpsf -- Maximum number of bins was increased from 100 to 1000. extractor -- A number of obscure bugs were fixed and efficiency was improved. fadmap -- Minor bug fixes. grppha -- Command "show group" was not checking the last channel if continued. mathpha -- Minor bug fixes; certain parameters were not being handled properly if set on the command line. xy2sky -- Fixed an infinite loop which occurred at the end of the tool run. ===== ROSAT ===== ===> New since v4.0 <=== (NONE) ===> Major changes since v4.0 <=== pcecor * New option to use either ground or inflight calibration data pcrpsf * Now able to ignore off-axis histogram data/WMAP extensions * (bug fix) All permutations of options weren't being handled properly pctcor * Number of supported AL K calibration maps increased from 8 to 13 ===> Bug Fixes and Minor changes since v4.0 <=== * The Rosat barycenter correction programs now take into account the June 30 1997 leap second. abc -- Bug causing loss of precision in times fixed. hriexpmap -- RA, Dec, and pixel size keywords given real values. hrifilt -- Several minor bugs fixed. hrirpsf -- Chatter output handled properly. pcfilt -- Problem handling very large data files fixed; eliminated spurious bad-data flags caused by uninitialized variables. pcsasscor -- Minor bug fix. rate -- Minor bug fix. ===== TIME ===== ===> New since v4.0 <=== (NONE) ===> Major changes since v4.0 <=== deadtime -- Restriction on number of deadtime intervals imposed by a fixed-size array was relaxed, and minor bugs fixed. fltime -- Maximum number of gtis was increased to 10000, and an error is now generated if this is exceeded. maketime -- Changed to allow unlimited length expressions. No longer writes zero-length gtis. mgtime -- Now uses a entirely different and much more robust algorithm for merging the GTI lists. Handles TIMEZERO keywords. ===> Bug Fixes and Minor changes since v4.0 <=== ===== VELA5B ===== ===> No changes since v4.0 <=== ===== XRONOS ===== ===> No changes since v4.0 <=== ===== XSELECT/EXTRACTOR ===== Note: the extractor was formally moved into the HEASARC package, so notes about the extractor may be found in that section of this document. xselect -- A major overhaul was performed to make it easier to add support for new missions and instruments. This should not affect users analyzing data from currently supported instruments. Anyone interested in adding a new mission or instrument should contact Keith Arnaud. (kaa@lheamail.gsfc.nasa.gov) In addition, xselect now properly handles filenames that include IAU-type position designations, unless the designations are at the end of the filename. Previously, these were mistaken for extension specifiers. Finally, many improvements were made to the xselect user's guide. Please check out the latest version at: http://heasarc.gsfc.nasa.gov/ftools/xselect/xselect.html ===== XTE ===== For complete details and up-to-date information on using FTOOLS v4.0 to analyze RXTE data, please refer to: http://heasarc.gsfc.nasa.gov/docs/xte/xhp_proc_analysis.html ===> New since v4.0 <=== correctlc -- Corrects lightcurves for variance in number of active PCUs. gti2xronwin -- Converts a GTI file (or extension) to a XRONOS-format time filter file, i.e., *.wi file. hxtdead -- Correct HEXTE spectrum or lightcurve for detector dead time. (REPLACES hxtdeadlc & hxtdeadpha.) Now corrects both archive and event modes; interpolates for small shortfalls of housekeeping; uses binsize larger than rocking interval. hxtlcurv -- Produces background-corrected HEXTE lightcurves. A [Perl] script to run hxtback, sa/seextrct, hxtdead, and lcmath; can run into problems with certain event modes, depending on the rocking period. pcadtlc -- Applies dead time correction to PCA .lc files. pcadtspec -- Applies dead time correction to PCA .pha files. sa2phaII -- Convert a Science Array (SA) file into a type II PHA file. sbmerge -- Combine concurrent single-bit (SB) mode data and create an output file in Science Array format. xsut -- GUI front-end to GNUtar for selective untarring of XTE data distribution tapes or data archive tar files. ===> Major changes since v4.0 <=== addshots: Removed case-sensitivity in input parameters. asmsrcupdate: (v2.2 -> v2.3) Script now correctly handles first file in incoming list not having an existing master. Also correct bug to properly rename new master files with _d1 designation. N.B. It appears that script does not properly handle a new source at the *end* of the incoming list (and likely never has.) Made output less verbose, and omitted one of the pause points. grosstimefilt: Script now capable of updating keywords TSTARTI, TSTARTF, TSTOPI, and TSTOPF. New parameter (-c) on the calling line so that users can answer "Do you want to save the file?". hxtarf: (v1.3 -> v1.4) Changes to code which reads on the on-axis ARF, including support for CALDB access. pcabackest: (v1.5 -> v1.5a) Using gain correction files not from caldb now works correctly. The user may specify a filename and an extension or the program will assume the default extension is 1. There is a new parameter "timeslop" which specifies by how much the filter file is allowed to be shorter than the Standard mode 2 file. The default is 32 seconds. This parameter is added in such a way that people with parameter files that do not refer to this parameter will just get the default. pcagainset: (v1.1 -> v1.11) Phase binner tools now work when obs_time=UNKNOWN. pcarmf: (v2.1.1 -> v3.1) PCARMF v3.1 now includes features which take account slow changes in the characteristics of the PCA. First, there is time dependence in the gain, with the gain changing by about 1% over two years. PCARMF uses the correct gain using the epoch 3 energy-to-channel values from the calibration file, the date of the observation, and hidden parameters in the parameter file. In addition, the amount of xenon in the propane layer is now slowly changing as well. Both this and the drift in the gain can cause odd features near the Xenon L edge. It is the intention of this version of PCARMF to minimize these features. Finally, the instrumental resolution is more nearly correct as compared to previous versions. PCARMF v3.1 uses as its energy-to-channel calibration file pca_e2c_e03v04.fits, which is available in caldb. pcarsp: (v2.31 -> v2.36) Omits copying of history keywords, as this is now done by addrmf and rbnrmf. Also modified the prompts and documentation. runpcabackest: Removed calls to SAEXTRCT saextrct: (v4.0 -> v4.0b) / seextrct: (v4.0 -> v4.0d) Two new (hidden) parameters: lcbinarray - Maximum number of light-bins to use gtiarray - Maximum allowed number of GTI array elements Bugs fixed which caused incompatibility with HEXTE tools (related to cases where GTIs overlap or abut each other). ===> Bug Fixes and Minor changes since v4.0 <=== tools such as fakelc, addsine, and addshots which use readlc() now truncate input arrays that are too large with a warning, instead of overwriting the array ends. pcaclrsp: Reformatted time output for chatter. recofmi: Bug fixed which caused crash in rare cases. sefilter: (v4.0 -> 4.0a) Bug fixed in E-token filtering. xpcaarf: Reformatted time output for chatter. xtederive: (v2.3 -> v2.6) Bugs fixed which caused (rare) crashes. ===== FV ===== ===> New since v4.0 <=== FV, had a stand-alone release in December 1997. The following items marked with an '*' are new since that version. * Entire extensions can be copied, cut, and pasted to/from the clipboard which can be saved to a new FITS file. - A spreadsheet calculator function allows you to create new columns by performing arithmetic on existing columns. - Tables can be sorted using multiple keys. * When sorting, rows with identical keys may be optionally deleted. * Table rows can be selectively deleted using a boolean expression evaluated for each row. * Individual columns or pairs of columns can be used to produce 1D or 2D histograms, respectively, and plotted in POW. * Selected rows or columns of an image can be averaged and their pixels plotted in a line graph. - POW now supports interactive manipulation of the image colortable (i.e. contrast and bias twiddling). - POW supports TrueColor displays and has more "colormap friendly" support for PsuedoColor display. See the documentation for the "-cmap" fv command line switch or the POW "Color" help menu item for details. - POW interprets standard FITS World Coordinate System (WCS) keywords, and optionally displays grid lines. * Once displayed in POW, the contents of graphs can be edited and multiple graphs can be merged into one. Coordinate information (labels, pixel sizes, WCS parameters, etc) can be modified as well. * POW can create, read, write, and graphically manipulate regions (boxes, circles, ellipses, polygons, lines, and points). (Data analysis of regions will be added in a later version.) * POW can "blink" a set of images or graphs in rapid sequence. A 3D image cube can be animated in this manner.