Command-line access to HEASARC data holdings
The HEASARC Archive can be more easily browsed through the web tools such as Xamin, but once a user has identified the datasets of interest, there are other ways to retrieve them. In particular, for automating the retrieval of large numbers of observations, directories, or files, command line tools remain useful. We summarize the options here. (For programmatic access to certain specific data products such as images, spectra, and catalogs, on the archive (and others!) through Python, please see our Python Virtual Observatory resources.)
Note that the situation changed in 2019 when all unencrypted access to the archive was discontinued. This page describes the current situation, based in part on our transition guide.
The HEASARC supports encrypted access through HTTPS or FTPS. For many cases, the protocols will be effectively the same, but in a few cases, the server may respond slightly differently. And different tools will behave differently depending on the specified address as described below. So there are a number of permutations.
To replace the old anonymous FTP, HEASARC will support explicit, passive-mode FTPS connections. Implicit FTPS is not supported, nor is active mode FTP. To replace the unencrypted HTTP, it suffices to change addresses to HTTPS (which will be done automatically in most browsers as well as in wget).
There are several command-line tools that can make use of these protocols such as wget and curl. Note, however, that they require different specifications of the address to get the protocol correct. Specifically, wget uses the ftps://heasarc.gsfc.nasa.gov address, while curl uses ftp://heasarc.gsfc.nasa.gov along with an additional flag to specify to use a secure connection. Alternatively, both tools can use the HTTPS protocol, which is how the Xamin tool generates download scripts.
Here is a summary of the available options:
(There is no straightforward way of getting an entire directory recursively with curl.)
wget -q -nH --no-check-certificate --cut-dirs=5 -r -l0 -c -N -np -R 'index*' \ -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/xte/data/archive/AO8//P80110/80110-01/.
If you have questions or encounter problems, please use the HEASARC's Feedback form to let us know.
HEASARC Home | Observatories | Archive | Calibration | Software | Tools | Students/Teachers/Public
Last modified: Wednesday, 13-Jan-2021 13:25:31 EST