Browse Browse Help Help
Latest Version
New/Updated Tables

The HEASARC Open Archive

The HEASARC archive and catalog resources are openly available for use by other archive sites. This document describes several ways that other archive sites can use our resources. Options include:

Install a mirror or modified copy of our Browse system
Query the existing CGI pages
Use the HEASARC batch interface
Access the archive through FTP and the catalogs through DBI.

The last option is especially powerful as it supports essentially arbitrary queries of the HEASARC catalogs and immediate access to all HEASARC data. Users can download copies of our software either for querying our site, or to help in building their own archive systems.

We begin by discussing the installation of our Browse service in some detail.

Tar files

Download this file, untar and follow directions in the README file.

Browse distribution file

Browse requires the Expat XML C library for the VizieR queries. For your convenience we include it here:

Expat XML library

The CPAN module can make the downloading and installing of required Perl modules easier. The README file has more information on this. The CPAN module and two modules it is dependent on are included here for your convenience. These modules are included in recent versions of Perl modules. They can also be downloaded at the CPAN site

Installing a Browse mirror

Browse is the HEASARC's primary interface to its catalog and archive resources. It comprises of a set of CGI scripts which progressively lead the user through mission, catalog and data selection. Browse uses no proprietary software, and is freely downloadable from the HEASARC. Browse supports distributed catalog and archive resources so that a mirror site can access local and HEASARC resources simultaneously.

The Browse system assumes the user has a relational database available. Communication with the database is exclusively through the Perl DBI interface. DBI modules are available to support all major database system including the free (for non-commercial use) MySQL and mSQL databases.

Users are free to modify the Browse code, however the HEASARC can provide only very limited support to users who wish to make more than cosmetic changes.

Overall Organization

The Browse installation files include the HTML pages, CGI scripts and executables used in the HEASARC Browse system. Some customization of these data will be required for remote access.

The HEASARC's Browse configuration uses directories specified by the installer for the

Location of HTML documents

Location of CGI scripts

Location of Perl libraries used in Browse

Location of executables used in Browse -- primarily the NED and SIMBAD name resolvers.

Location of class files used in HEASARC Java applets.

Customizable Files

There are several files that remote users will want to look at for customization:

/www/htdocs/cgi-bin/W3Browse/lib/Browse/Config.pm

This module controls a number of configuration options for Browse, including the directories in which various files will be found. Please read the comments in this file for directions on customizing it.

w3browse.pl

This is the HEASARC's home Browse page and is located in the cgi-bin script directory. Other sites may wish to customize its appearance and to include links to sites appropriate for their usage.

w3header.html

This file in the HTML document directory contains html for the header part on w3browse.pl, version-history.html and w3browse-help.html pages.

w3browse-form-tables.html, inc_*html

These files in the HTML document directory control the apppearance of the main interface.

databases.dat

This file in the cgi-bin directory indicates the databases that Browse will query. The first line describes the default database that Browse will use for all metabase queries. A line is also required for all remote databases in the system. The databases.dat file consists of four fields. The first field is the database name used in Browse. This corresponds to the database_location field in the Browse ZZGEN metabase table. The second entry is the DBI name for the database. The third and fourth entries are the user and password to be used in connecting to the database.

Database Connections

Communication with databases is the Perl Database Interface (DBI) protocol. A Browse site will need to install both Browse software and the DBI software. The DBI is available at the Perl CPAN site: www.cpan.org.

The DBI comprises a database independent layer -- the DBI itself, and database dependent modules called DBD's. DBD's are available from CPAN for all popular databases including Sybase, Oracle, Informix and Ingres. To access the database, a Perl script makes a DBI connection. The DBI connection name includes the type of the database and so DBI calls the appropriate DBD to actually make a connection to the database. For Sybase and Oracle a direct connection to the database port is made. Subsequent DBI calls can use the existing connection.

Detailed instructions for Perl modules needed for Browse

We include here a fairly extensive description of how to install the Perl modules needed to use Browse. Not all may be needed for every installation. Some Perl modules depend upon others so that the order in which we install features does matter. We have included versions of some of the relevant Perl modules in our distribution library, but you may wish to look at the CPAN site for more recent versions.

We suggest the CPAN modules be installed in the following order:

Net::Daemon
Storable
PlRPC
DBI
URI - needed for LWP
HTML::Parser - needed for LWP
MIME::Base64 - needed for LWP
Digest::MD5 - needed for LWP
Net::FTP - needed for LWP
LWP - needed for querying VizieR tables and for viewing FITS files
XML::Parser - needed for querying VizieR tables
Filesys::Diskspace - optional

Note that some modules want the h2ph command to have been run on system include files. I.e., the user should have done something like: cd /usr/include h2ph *.h */*.h */*/*.h

The DBI module creates the DBD entry for proxies. If you are only going to talk to proxy servers, no further DBD's need to be included. If you are going to talk to a native database, or if you are going to install a proxy server yourself, then DBD's for the databases to which you are going to connect will also have to be installed.

Typically you will need to set appropriate logical variables to install these DBD's correctly. Copies of Sybase and Oracle DBD's are included in the distribution directory. E.g., to add in the DBD for Sybase, the environment variable SYBASE must be set to the SYBASE installation directory, and the LD_LIBRARY_PATH must be set to the directory $SYBASE/lib. These environment variables will also need to be set whenever a Sybase DBD is used within a Perl script.

For a given Perl module the sequence of steps to install the module might look like:

gunzip DBI-1.06.tar.gz
tar xf DBI-1.06.tar
cd DBI-1.06
[The user should look for README files to see if there are any special requirements for the system being used. This may also indicate that some other module needs to be installed before the DBI. ]
perl Makefile.PL
[This step creates the Makefiles which will build and install the extension. It should also check to see if any other modules need to be installed as prerequisites.]
make
make test
make install

Setting up a proxy server

A proxy server allows users to connect to a database that they could not otherwise access. A database may be restricted due to security concerns, or a remote site may not have the software libraries installed at their site which are necessary to talk directly to the database. Starting a proxy server is very easy. If the DBI module has been installed as described above (i.e., with the other modules installed first), then a dbiproxy command should have been created. Just enter dbiproxy --localport=#### & where #### is a desired port number to start the proxy server. This proxy server will then be able to relay requests to any other database that it can connect to. E.g., at the HEASARC we have a database running on the machine dbsrv.gsfc.nasa.gov. To access it from the HEASARC we would open it using a DBI name of dbi:Sybase:SYBASE where SYBASE is the name that at the HEASARC have given our database (not very original I know). If ISAS had the appropriate Sybase libraries it could link the Sybase DBD directly into its version of Perl and could call this database directly. However if ISAS does not have Sybase, they must call a proxy server. At the HEASARC we start a proxy server using the dbiproxy command on the dbsrv machine as:

dbiproxy --localport=9119

Now ISAS can access our database using the DBI name

dbi:Proxy:hostname=dbsrv.gsfc.nasa.gov;port=9119;dsn=dbi:Sybase:SYBASE

Note that the proxy server supports persistent connections, so that we can still access the database quite efficiently.

HEASARC Metabase tables

The Browse system relies upon the existence of 5 metabase tables. The ZZGEN table contains information about each table in the database. The ZZPAR contains information about the parameters of each table. The ZZEXT table contains extended information about tables. The ZZDPSETS table describes the kinds of data products that are associated with a given table. The ZZDP table describes data products available in the archive. This metabase is described in HEASARC documentation. Users may wish to look at dumps of the HEASARC tables to understand how to populate these tables. Note that the metabase describes all tables that a Browse system knows about even if the table data itself is stored at some other site. I.e.,
if ISDC wishes to query tables at both ISDC and HEASARC, the ISDC metabase must contain descriptions of all of these tables.

Setting up the Name Resolvers

Two Perl DBM libraries are used to cache results of name resolver requests. These are located at the HEASARC at

/www/htdocs/cgi-bin/W3Browse/dbm

which directory is created by the CGI tar file.

You need to initialize these files, or queries for named objects will fail.

Create four empty files:

NED.dir, NED.pag, SIMBAD.dir and SIMBAD.pag

These will need to be writable by the user ID under which the CGI scripts run. Let us know if you'd like current copies of our tables -- caching NED and SIMBAD queries may be particularly desirable from Japan unless there is a local mirror. Currently, the Simbad query program uses the US mirror. If your connections to Europe are better you may wish to reset it to use the main Simbad site.

Java Plotting Tool

The Browse java plotting tool allows users to plot results by running java applets or applications. The application(server) side requires a version of Java 1.1.x installed in your web accessible area.

You would need to update the variable baseUrl in the /www/htdocs/classes/heasarc/w3browse/HeasarcQuery.java file with the correct url reference. Then compile it using 'javac HeasarcQuery.java'.

It is best to recompile the class files in /www/htdocs/classes with your version of Java 1.1.x. If you wish to speed up the client-side downloads, you may create w3browse.jar and ptolemy.jar files using commands:

cd /www/htdocs/classes
jar cf w3browse.jar ./w3browse
jar cf ptolemy.jar ./ptolemy

Getting the Server-side Application to work
Here are the steps involved in getting the application side to work:

Obtain a version of JDK1.1 from http://java.sun.com/cgi-bin/java-ports.cgi suitable for your particular platform and install it.
The $::jbin and path needs to be set in the Config.pm file.

Primary Pages

There are many documentation pages, but the key pages for users are the following (assuming the HEASARC's Web organization). The web form names are listed and then '->' indicates the name of the executed script after the form is run.

Browse Home Page
   W3Browse/w3blaunch.html  (or just W3Browse)
   
      Allows the user to select from several interfaces.
      
Main Interface
   cgi-bin/W3Browse/w3browse.pl    ->   w3table.pl
   
      Allows users to select one or more missions of interest.
      
Query page
   cgi-bin/W3Browse/w3table.pl     ->   w3query.pl
   
      Format a query against one or more tables.
   
Query Results page
   cgi-bin/W3Browse/w3query.pl     ->   w3hdprods.pl, w3cross.pl
   
      Get the results of a query.  Ask for 
      cross-correlations of results or data products.
      
Data Products page
   cgi-bin/W3Browse/w3hdprods.pl   ->   w3tar.pl

      Show the available data products.  User
      can create tar file of desired products.
      
Tar page
   cgi-bin/W3Browse/w3tar.pl
   
      Create and download results tar file.
      

Tables cross-correlation page
   cgi-bin/W3Browse/w3fullcross.pl
   
      Cross-correlate two tables.

Results cross-correlation page
   cgi-bin/W3Browse/w3cross.pl
   
      Cross-correlate query results with another table.

Using Existing CGI Scripts

A quick way for remote sites to get up in using the HEASARC resources is to use a command line browser or HTTP access library (e.g., the Perl LWP library) to query the HEASARC's Browse Web pages. This method is not recommended, since it typically relies upon the detailed format of Browse queries and the returned results.

Using the Batch Interface

The HEASARC's Batch Interface to the catalog system was designed to provide a mechanisms for users to query HEASARC catalogs from a command line environment especially scripts. It can easily be adapted to be used within CGI scripts. There is little explicit support for access to the HEASARC's archive resources.

Open access using DBI and FTP

All of the HEASARC's catalog and archive resources are directly accessible to the user. The HEASARC provides a DBI inteface through which outside users can send arbitrary queries to the HEASARC database. All archive files are immediately and individually visible through FTP and HTTP protocols. The DBI interface to the HEASARC catalog is discussed above.

Sites going this route may want to understand the HEASARC catalog structures and metadata.

National Aeronautics and Space Administration

Goddard Space Flight Center

Sciences and Exploration

Search: