The Xamin Guide v1.2

Introduction

What is Xamin?

Xamin can be used through a web interface designed as a Rich Internet Applicaiton (RIA) and also through a command line interface. All facilities are available through both interfaces.

Documentation Overview

The Xamin Guide

This document itself provides a general overview of the Xamin system with particular emphasis on how the system is built and how it can be used. Users interested in an overview of the general capabilities may find this document more accessible than the detailed interface guides. This is the primary document for administrators since it describes in detail the steps for installing and maintaining the Xamin system. Developers may find this document less useful since it discusses the code in only fairly general terms, but the code overview may be a helpful starting point.

This web guide -- designed to be used from the Xamin Web environment -- describes the Web interface in detail.

The Xamin CLI User's Guide

This document describes the use of the Xamin CLI interface. The Web interface is based upon the CLI and the CLI document spends a bit more time on the details and consequences of certain settings, so this document may be useful for both Web and batch users.

The Xamin Plotting User's Guide

This document describes the powerful plotting capabilies.   Xamin's plotting interface is build over the plot.ly library.

The Xamin requirements document.

Current implementations

    https://heasarc.gsfc.nasa.gov/xamin
        

Migrating from Browse to Xamin

Querying all HEASARC holdings

This works essentially the same as before. Just enter a position and/or time and and start a query. You get back a table with the number of hits for each HEASARC table. Click on the table icon to query a specific table with the specified constraints.

Querying a specific table

In Browse if you wanted to query a specific table you could go to the index page or first select a mission and then the specific table. In Xamin you enter the table in the Tables Explorer quick entry box at the top of the form, or you can browse the the table entries in the Tables Explorer hierarchies. You can also search for tables using a keyword search of their documentation. Any of these can be used to select a table.

Querying all tables from a given mission

This was the default mode for Browse when the user checked a mission box. In Xamin, to select all of the tables for that mission just open the Tables Explorer/mission hierarchy and click on the icon to add all of the tables for that mission. Then use Find matches in... Selected Tables to search all of the tables in the mission for the selected time or positions. You can add tables from other missions, or delete some of the tables before querying if you like. You can then query the selected tables for position or time constraints.

How do I do a query by parameters?

Enter the table name the use the Parameters Explorer to add parameter constraints. You can add a constraint for any parameter by clicking next to it. Note that you can use fairly general SQL expressions in the constraints and these can involve table parameters.

How do I do cross-correlations?

If you have 2-4 tables selected, you can do a correlation of them just by clicking on Query to...Correlate. Spatial cross-match constraints will be added automatically, but you can override or modify these to meet your needs.

Can I search the metadata for tables as I could in the keyword entry interface?

Yes. Use the Tables Explorer keyword search.

How do I monitor a mission for new observations of the target I'm interested in?

Xamin has a broad capability to check if the results for any query have changed. First get a HEASARC services account. Login during an Xamin session and run a query that finds observations you are interested in. Use Query to...Save to save this in your account. Bring up the summary of your saved tables with theSession/User Tables--Info menu. If you click on the Frequency column next to your saved table you should get a pane that allows you to ask to be informed whenever the result of this query change.

System Elements

Interfaces

Web Interface

CLI Interface

Queries

Single Table Queries

Discovery queries

Correlations

All tables can take part in correlations. However remote tables must be downloaded to the HEASARC before the correlation can take place. So in practice correlations of remote resources can only be done if the tables are of modest size or if constraints that can be passed to the remote site can restrict the size to a reasonable number. It may be impractical to query remote tables where the size of the downloaded data is more than ~10,000 rows.

In the web interface spatial correlation constraints are assumed and created for the user -- they can be deleted if not desired. In the command line version the OFFSET keyword is used to specify positional correlations, but the user must include this explicitly.

Temporal cross-correlations can be made as simple constraints on the time columns of the two tables. At the HEASARC times are stored in MJD's, so to ask for observations within a day of one another we can add a constraint of the format

     b.time-1 .. b.time+1
        

     constraint='a.time between b.time-1 and b.time+1'
        

While any number of tables can be correlated in principle, queries tend to become very slow when large numbers of tables are involved. The web interface imposes a maximum of four tables in a correlation, no limit is specified for the command line. This limit can be gotten around if necessary by breaking up the correlation into a series of correlations and saving intermediate results as tables.

Anti-correlations

• A query is performed on all tables of the query including all of the non-group constraints (e.g., all of the constraints that would go in the WHERE clause in the SQL). The fields returned in this query are row identifiers for the non-anticorrelation tables of the query. E.g., if we are querying tables a, b, c, and d with an anticorrelation on table d, then we apply all of the constraints on the query (including constraints on table d) and find out the row tuples for tables a,b and c. These tuples are stored in a temporary table. E.g., one of the results of this query might be that row 5 of table a, row 6 of table b, and row 7 of table c are found, so we save a tuple (5,6,7).
• A second query is now performed where all query constraints that involved table d are removed and a new constraint is added that the any tuple of rows that was found in the first query is excluded. The output fields that the user asked for, and any group constraints (constraints that would be put in a HAVING clause) are added back into the query. The results of this query are returned to the user.

Note that Xamin does not do careful analysis of the input constraints to determine if pieces of an input constraint can be included in the second query. E.g., compare the constraint

     constraint='d.exposure>1000 and c.exposure>1000'
        

     constraint=d.exposure>1000,c.exposure>1000
        

Autocorrelations

When doing an autocorrelation it is usually necessary to make sure that you don't return a row matching itself. A simple way to ensure that is to add a constraint of the form

       constraint='a.id != b.id'
        

Data Products

There are two fundamental types of products:

1. Mission archival products represent the data holdings in the HEASARC archive. These are the mission data from more than two dozen NASA and non-NASA missions. For many Xamin users querying observation catalogs is just a step in finding and locating the datasets of interest to their research. These products and be retrieved and downloaded using Xamin.  These products are defined in mission data product files.  All data products for a given mission are normally defined in a single XML file.
   
2. Link products are URLs that point to data that are related to a specific row, but are not part of the HEASARC archive proper. We distinguish several types of link products.
   • Linked tables return rows in a linked table that are related to the row in the initial table. E.g., each row of the SwiftMaster table links to the associated rows in the Swift instrument tables (BATMaster, XRTMaster and UVOTMaster). In turn the instrument tables link to the mission master table. Table links can and often do span missions. Table links are provided between many of the major observation master tables, e.g., linking each Chandra observation to nearby XMM observations. Following a table link executes a new Xamin query. Such link products are automatically generated from the Browse zzlink.tdat file which is updated whenever new table links are included.  A TableLinks.xml file may preserve the current table link definitions.
     
   • Bibliographic links provide links to papers using HEASARC observations. A master bibliography table, ZZBIB, indicates which observations have been used in which table. The table has been collected manually and does not cover all of the literature for all missions but covers much of the HEASARC's holdings.  Since these links are pointed to from the zzbib table, they are defined in the PointedProducts.xml file.  This needs to be updated whenever entries from a new table is added to ZZBIB.
     
   • Dynamic service links use positional information from each row to query other services. Currently the HEASARC provides service links to NED, SIMBAD and SkyView. Unlike other products these are available for any table without needing to be specifically associated with a table. The mechanism used for positional service links is not restricted to positions. If a remote service can be invoked by specifying a value and the metadata for a table indicate it has a field or fields that have appropriate data, a dynamic link can be made.

Downloading archive data

As data products are collected they are added to a shopping cart. The shopping cart can contain products from multiple selections and from multiple queries. Users can also download individual data products directly (or in the case of directories open a web link to the directory). Link products cannot be added to the shopping cart -- they generally involve data that is intended to be viewed in browser or which are found at remote sites.

At any point in an Xamin session users can download the data in their shopping cart. They can download individual products one by one from the shopping cart, but if there are more than two or three products users may prefer to collect the data products together. A tar file of all of the products in the cart can be created. This tar file is created dynamically so the download should start almost immediately.

Whenever a significant amount of data is to be downloaded we recommend that a data download script be created rather than directly tarring the data. Scripts can be created which use either the WGET or CURL commands both of which are widely available. Scripts are much more easily restarted and allow the users complete control of the data download process. The script can be displayed and copied via striping or it can be downloaded explicitly.

In the command line environment a query can be used to build a download script to download archival products or to provide a list of links to link products.

Setting Up Data Products

• Product descriptions are table independent: they describe the content of a product and where it is located (or how to construct its URLs) in terms of a set of variables. A product may have other products as child products. A child product inherits the variables of its parent. Often products specify a file template which is used to match against the file system relative to a root location. A root variable will point to some directory associated with a specific observation.
• The TableLink metadata  sets up a link between a specific table and a product. Many tables can link to the same a product. The table links describe how the values of the variables in the product definitions should be created based upon the columns in the table the product is being linked to. They can also specify a boolean expression that must be satisfied if the product is to exist. This allows for conditional products. E.g., we might wish to link from a table of objects to different linking tables depending upon the classification of the object. [There is an unfortunate confusion in terminology between this use of table links, i.e., the links between a table and any products, and a particular kind of product, where we want to link one row of a table query to a query of some other table.]
  

User Sessions

Capabilities

On-line users can also save the current query configuration at any time into a named configuration. This configuration can be restored at any later time. Note that the configuration only affects the Query window, not any of the other Xamin windows. If a configuration is saved with the name default, then auser can automatically restore this configuraition in subsequent logins. To change the default configuration the current default must first be deleted.

Output formats

ASCII

VOTable

VO Protocols

Excel

FITS

Table Sources

Local Tables

User Tables

Uploaded Tables

External Tables

Sample Queries

Discovery Queries.

Examples:

• What HEASARC tables have information near 3c273?
• Does the HEASARC know of any observations made on June 6, 2007?
• What X-ray data does the HEASARC have on Cen-A in 2005-2006?
• What ROSAT, Chandra, XMM and ASCA observations does the HEASARC have on 3c273, Abell 1656 or 10 15 06.2, +14 25 19.8?

Simple Single Table Queries

Examples:

• What does the XMMMaster table look like?
• What observations has Chandra made of 3c273 or 3c279?
• What did XTE observe on 2004-10-10?

Complex Single Table Queries

Examples:

• What observations has ROSAT made of 3c273 that had exposures > 10,000 seconds?
• What are the longest XMM observations?
• What is the average flux of RASSFSC sources as a function of galactic latitude?
• What high-galactic-latitude sources in the XMM Serendipitous Source Catalog have fluxes more than 10 times the flux error?
• For each source observed by ROSAT what were the total number of observations and the total exposure time of all observations of the source per instrument?

Cross-correlations

Examples:

• What Chandra observations have been made of objects in my uploaded source list?
• What targets have been observed by both ROSAT and Chandra?
• What targets have been observed by ROSAT with an exposure of at least 40000 seconds and Chandra with an exposure of 10000 seconds?
• What targets have been observed by Chandra with an exposure time greater than the ROSAT exposure time?
• How many times has each Messier object been observed by Chandra?
• What is the x-ray/radio flux ratio of objects seen in both the XMMSSC and FIRST catalogs?
• Which of the objects in my source list have not yet been observed with Chandra?
• What are the brightest objects classified as AGN in the WGACAT catalog that have not been observed by Chandra?

External Queries

Examples:

• What are 100 nearest USNO-B sources near the north Galactic pole?
• Are there Chandra observations of black holes with masses greater than 100 solar masses? (E.g., J/ApJ/667/131 is a table of black hole masses).

Monitors

Examples:

• Let me know about new XMM or Chandra observations of Abell 496 or Abell 1656.
• Are there any new Chandra ACIS observations with exposures longer than 10,000 seconds?
• Is there any new information for any tables with positional errors less than 1 degree on TY Pyx?
• Maintain a list of distinct PIs and the number of observations for Suzaku.

Upload Queries

Examples:

• Which elements of my source list are at high Galactic latitude?
• Given a CSV file with RA,Dec and Magnitude, find the difference between these magnitudes and magnitudes in the USNO B catalog?
• Given a VOTable with a set of stellar positions, find the objects where the offset with Hipparcos positions is larger than 20" (assuming that any offset >20" is a misidentification).

Installation

This section describes the steps in creating the complete Xamin data access system. The discussion of each step includes both the general issues that need to be resolved by an implementor in their system, and the specific choices made at the HEASARC.

This section describes a complete installation of the Xamin server. Users who only wish to be able to use the Xamin from their CLI environment need only download the very small users.jar file from the Xamin distribution directory (e.g. http://heasarc.gsfc.nasa.gov/xamin/distrib/users.jar).

Installing and populating the database.

Xamin is designed to use a PostgreSQL database. Some PostgreSQL specific features are used in the software to populate the system and in the query system. The keyword search interface uses PostgreSQL features that paralleled in some other systems (e.g., SQLServer), but may not be generally available. Postgres specific features of the system are called out

Postgres is freely available at:

   http://postgresql.org
        

While one click install downloads are available, these generally require superuser privileges on the destination machines..

Installing the database

The tar file should be unpacked with

      > tar zxf postgresql-X.Y.Z.tar.gz

        

Within the postgres-ql-X.Y.Z directory

    > ./configure --prefix=desired_postgres_directory
        

    > ./configure --prefix=/local/db/postgres
        

Then execute:

     make
     make install
        

The current Postgres operational machine is installed on the DBMS1 machine.

The Postgres server is ideally run under a special user account but may be run under a normal account. The special account can be created using normal system procedures.

Initial configuration.

To initialize the database first create the directory in which the database data is to be stored. This directory should be owned by the same user as will run the server.

    > mkdir directory_for_data
        

    > initdb -D directory_for_data
        

To create an initial superuser account and password you may wish to include the -U and -pwprompt or -pwfile=file arguments to the initdb command. The -U argument specifies the name of a superuser account to be created for the database (e.g., postgres). The -pwprompt argument indicates that the command should prompt for the password for the account while the -pwfile argument gets that password from a file.

The output from the initdb command indicates how one can start the database server. This command should be put in the startup scripts for your database machine (but is not currently for the HEASARC).

Once the database has been initialized some small amount of customization of the configuration files is desirable. These are found in the data directory mentioned above.

• The pg_hba.conf file should be updated to allow remote access. This can be configured in a number of ways.
• We have also made the following changes to the postgresql.conf file:

     shared_buffers = 500MB
  
     temp_buffers   = 8MB
     max_prepared_transactions = 5
  
     work_mem = 16MB
     maintenance_work_mem = 64MB
     max_stack_depth = 8MB
                  

  All of these lines are updated from the values in the default configuration which is sized for a small machine.

Postgres accounts

A second account, e.g., xamin, may be defined as the owner of the xamin database with all associated privileges, or this can be done under the postgres account. Using a separate xamin account is preferable since this account need have no privileges outside of the Xamin database. The account will be used in many of the internal operations scripts, but is only invoked by external user actions when new 'user accounts' are created and destroyed and the associated schemas need to be created or destroyed in the Xamin database.

The webuser account is the account that external users normally invoke when accessing Xamin. The webuser account has the privileges to read all non-system tables and to create and update tables in user schema where the tables associated with user sessions will be supported.  Note that all users share the same actual database accoun but cannot read other users' data since a password is required to access session data.

The names and passwords of these accounts are needed in the [heasarc|ops|dev].settings files found in the Xamin WEB-INF area. Generally the ops/dev settings files will use the xamin account which is also the privileged account in the heasarc.settings. The webuser account is used as the non-privileged account in the heasarc.settings.

To create a user in Postgres use the CREATE ROLE SQL command or the createuser script provided in the Xamin user area. E.g.,

                create role xamin with login encrypted password 'password';
                create role webuser with login encrypted password 'another';
            

The Q3C and PGSphere libraries

            make
            make install
        

After the code has been built log into Postgres as the Postgres superuser and enable the extension for the HEASARC database using the command

             create extension q3c;
        

The PGSphere library must be downloaded. The latest version is currently at https://github.com/akorotkov/pgsphere. After copying it into the contrib area and building it similarly to

Preparing for data ingest.

Before starting data ingest, an installer will need to download the the needed jar files. Jar files are modified zip files which are used to store Java classes. However they can also be used to store other kinds of files including data files used for running programs. The Xamin jar includes a number of utility programs (mostly written in Perl) that need to be unpacked to be used. The Xamin war file is created in building a full distribution of Xamin. This war file (a jar file containing the complete Web application) is currently created using NetBeans. To unpack the war, copy it to the directory in which you want to unpack it and

   >  jar xf xamin.war
        

0
   > java -cp $XAMIN_JARDIR/Permissions.jar permissions.PermissionRestorer $XAMIN_ROOT/.. < $XAMIN_ROOT/../executables.list
        

Initializing the database.

The initialize command in the .../ops/ingest directory will perform many of the tasks noted below: defining the HEASARC database, populating it with HEASARC tables, creating the table documentation and master tables, setting up the areas for supporting VO TAP queries, and ingesting data products. More details on these steps are given below.

Many of the environment variables used in operations scripts are defined in the .../ops/daily/cronenv.sh file which is used to initialize them before they are run. The settings in this script should be carefully examined and may need to be updated. The environment variables include:

APP_ROOT

Defines the base directory of the overall Xamin web application.

XAMIN_ROOT

Defines the base directory of the Xamin operations scripts typically, $APP_ROOT/WEB-INF/classes/ops.

XAMIN_JARDIR

The location of the JARS used within Xamin, typically, $APP_ROOT/WEB-INF/lib.

LOG_ROOT

A directory containing subdirectories in which which system commands outputs are placed, notably the system operations logs. This does not log user requests to Xamin.

POSTGRES_ROOT

The root of a Postgres installation. This is mostly used to find Postgres executables, e.g., $POSTGRES_ROOT/bin/psql

XAMIN_LOGDIR

The directory in which system actions are logged, usualy $LOG_ROOT/logs

XAMIN_BACKUPS

The directory in which user backups are placed, usually $LOG_ROOT/bck

XAMIN_TMPDIR

A directory in which system commands can create temporary files, usually $LOG_ROOT/tmp

PGHOST

The computer where the Postgres database is to be found

PGUSER

The user account under which requests are to be made. Normally this will be the postgres or xamin user for system operations. This will be set in the settings file for external Xamin access.

PGDATABASE

The database to queried. Normally this will be the heasarc database.

TDAT_DIR

The directory (on the TDAT_HOST) where TDAT files are stored.

TDAT_HOST

The host machine on which TDAT files are stored.

HEASARC_SETTINGS

A pointer to the settings file to be user. The settings files are generally used to convey information to Java programs. Operations scripts normally use the dev.settings or ops.settings (depending upon whether this is a development or operational installation), while the web application generally uses heasarc.settings. All of these files are included in the $APP_ROOT/WEB-INF directory.

Creating the database

Postgres allows for multiple databases to be supported by a single Postgres installation. Each database can have multiple schemas and each schema multiple tables. Cross-database queries are not supported, but cross-schema queries are. All tables used within Xamin should be placed within a single database which may now be created either by using the psql command (the CLI interpreter for the database), or using the createdb command. E.g., using psql we might have:

    > psql
    prompt=# create database heasarc;
    CREATE DATABASE
    prompt=#
        

    prompt=# alter database heasarc owner to xamin;
        

Metadata and feedback table creation

The metadata table, metainfo, needs to be created before any other tables can be read into the system. The

    buildmeta.csh
        

    buildfeedback.csh
        

Copying tables

   > runfullcopy
        

The runfullcopy command uses other scripts in the directory. It calls fullcopy after setting up logs. This calls copyit for each matching TDAT file. The real workhouse inside copyit is pgcopy.pl which parses the TDAT files and creates the SQL to ingest the table. This is then sent to the Postgres psql to be executed.

The procedure populates both the tables themselves and the metadata regarding the tables. However it does not populate the data products related to a table.

In addition to populating the science tables, the runfullcopy command updates the documentation tables within Xamin. Xamin keeps a copy of all HEASARC and Vizier table documentation in the database. These can either be fully refreshed or updated depending upon whether an argument is specified in the command. The updateDocs.pl script is used by runfullcopy to find and ingest documentation.

Creating and updating the documentation table

This resulting SQL file should be run and then the fixdocs.sql script should be run. This last script adds in a bit of HEASARC metadata to the HEASARC descriptions, populates the TSQUERY column in the tabledocs table and gets rid of descriptions of HEASARC tables that are not present in the database.

Ingesting HEASARC product metadata.

The metadata for HEASARC products is ingested using the updateProducts command in ops/products, specifying a directory with the product definition files for each mission. E.g., as the code is distributed if the user makes the directory specified above the default directory then the command

   updateProducts missions
        

The HEASARC has a products file for each mission and special products files for table links, external links, and bibliographic links that should be individually processed:
    updateProducts PointedProducts.xml
    updateProducts PosProducts.xml

A cron job which looks at the zzlink.tdat file updates the linked table dataproducts automatically.

The updateProducts command only updates a single Xamin database.  If there are multiple databases that need to be updated (e.g., backup and test databases), then it
may need to be run multiple times to update the other tables.  See the documentation for this command if needed.

Building master tables

    buildFullMasters.pl
        

Master tables are views of two underlying tables a _prime table and a _bck. Creating the master tables is fairly expensive because we create a clustered index on fairly large tables. We do not wish to recreate them and regenerate the index each time we update a table. After buildFullMasters.pl completes, the _bck tables are empty. As tables in the database are updated, part of the update process is to add the position/time information from those tables into the corresponding _bck tables. Another table keeps track of the tables that have been updated since the last time we ran buildFullMasters.pl. When a user makes a subsequent query of the master tables, we look in the _bck table for information about science tables that have been updated, and in the _prime table for information about unchanged tables. Since only a small fraction of the tables get updates, the _bck table is much smaller than the _prime table. Thus even though it is not as efficiently indexed, master table queries can run quickly.

Periodically we run buildFullMasters.pl to bring the data from the _bck tables into the primary tables. Since this takes a couple of hours, the approach is to build up the new versions of the master tables using temporary table names. Only when the new master tables are complete do we drop the previous version and rename the new tables to the standard names. This takes seconds.

Preparing for VO Access

The heasarc.settings file.

The heasarc.settings file is used to set up the defaults for the Xamin system. The file is normally found in the WEB-INF subdirectory of the Xamin web application. Below we provide an annotated version. Other installations will need to adjust the values to meet their requirements. End users should generally not need to worry about it. It is not used on the client side of the Xamin batch interface.

# The default database
# This will be used as the suffix in the jdbc_xxxx settings
database=postgres

# The JDBC parameters for Postgres

# This is probably the same for everyone.
jdbc_class_postgres=org.postgresql.Driver

# The host and database name will need to be updated.
# The port is the Postgres default port.
jdbc_url_postgres=jdbc:postgresql://db.host:5432/postgres
jdbc_user_postgres=dbuser
jdbc_pwd_postgres=password

# Nominal hosts versus actual host names.  We want to
# be sure to translate any hostname we get into a valid
# host.  So if we get heasarc we want the full path
# heasarc.gsfc.nasa.gov.  If running on a load balancer
# then we want to translate the actual server name
# that we are running on, to the name that we want to
# publish.
heaweb1=heasarc.gsfc.nasa.gov
heaweb2=heasarc.gsfc.nasa.gov
heaweb3=heasarc.gsfc.nasa.gov
heaweb4=heasarc.gsfc.nasa.gov
# What to translate the host "localhost" to.
localhost=heasarc.gsfc.nasa.gov

# These are settings that the user is not allowed to change.
fixedSettings=database,jdbc_class_postgres,jdbc_url_postgres,jdbc_user_postgres

# The User account for non-privileged access
standard_user=dbuser


# Is there forwarding to a TOMCAT server.  This happens only
# for the nominal released and operational versions, not development versions.
# When we are constructing a local URL from the elements available in the
# server environment we may find that the servlet is running on port 6080,
# but we want to publish the original port 80 for the Apache server.
ForwardedPort=80,6080

# The current version of Xamin.
# The first digit is incremented with major releases.
# The second digit is incremented when there is a significant change to user functionality.
# Letters are appended to show bug/documentation fixes that do not appreciably
# change end user functionality.
Version=1.3g

# The following two fields are place holders that we fill in during the installation
# procedure with appropriate versions for the HDB and Xamin projects that are used
# in building the Xamin web application.
HDBVersion=0
XaminVersion=0

# This is replaced by the current date in the installation procedure.
ReleaseDate=2011-01-01


# This is a relative or absolute URL to service that should be invoked
# to request a user login.  Note that Xamin itself does not manage user logins.
# While a user is logged in, all Xamin requests use the login service
# to validate that the indicated user is logged in to the current session.
LoginServlet=/srvlogin

# This is a pointer to the location of the HEASARC hera service.
HeraServer=herasrv.gsfc.nasa.gov
HeraPort=60000
DftHeraName=AnonHera

# These are the E-mail addresses for administrators who will receive notifications
# when Xamin users submit feedback.
feedbackRecipients=user1@xxx.gov,user2@xxx.gov

# These provide a skeleton for the structure of download requests.
wgetcommand="wget -q -nH -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks "
curlcommand="curl -s --remote-name "

# User download requests using the TAR download feature are constrained
# to the tree below.
DownloadPath=/FTP
        

Preparing the Web server.

Customizing TOMCAT

<Valve className="org.apache.catalina.valves.AccessLogValve"
      directory="logs" prefix="localhost_access_log." suffix=".log"
      pattern="common" resolveHosts="false"/>
        

   <VirtualHost>
        ....
        ProxyPass        /xamin http://localhost:nnnn/xamin
        ProxyPassReverse /xamin http://localhost:nnnn/xamin
        ....
   </VirtualHost>
        

Setting up the Web applications

Xamin is intended to run in a servlet environment and at the HEASARC is developed using TOMCAT. To install Xamin in TOMCAT simply copy the xamin.war file to the TOMCAT webapps directory and restart TOMCAT. This should automatically install the system. Installers may need to update the heasarc.settings file in the webapps/xamin directory.

     mkdir webapps/xamin
     chdir webapps/xamin
     jar   xf some/path/to/xamin.war
        

The Xamin war file contains the full Xamin distribution including operational scripts. Scripts are generally found under the xamin/WEB-INF/classes/ops directory. A site may wish to move these operational scripts to other director
ies, however the WEB-INF area and its subdirectories are invisible to the outside world so this area can be used without revealing operational details to the public.

Installing new versions of Xamin.

In practice we use the xaminInstall.pl procedure to install new versions of Xamin.  This includes multiple steps and provides detailed feedback on each of the steps in the process.  It allows us to use a common installation procedure when there are multiple developers who may update Xamin.  The steps in the install are:

Step 0: Get arguments and confirm we are on right machine.
Step 1: Create a temporary directory to place the data content in.
Step 2: Unjar the Xamin war into the temporary area.
Step 3: Copy the needed JavaScript, JSP and HTML files from the  temporary area into the ExtJS compilation area.
Step 4: Use the ExtJS CMD commands to create the compressed JavaScript files
Step 5: Copy the compressed JavaScript files back to the temporary area.  This will update the xamin.jsp and single box html files  and add the compressed JavaScript files.
Step 6: Customize the settings file.
Step 7: Fix and update other files in the temporary area before release.
Step 8: Rename the current operational version to the backup name (prod and test)
Step 9: Rename the temporary area to the desired name.
Step 10: Restart the servlet.

There are a number of checks made to ensure that the size of the war file and the number of libraries and executable files is reasonable and that the servlet successfully restarts.  Steps 3-5 relate to an optimization of the JavaScript used in the Xamin web application.  The web client is divided into almost 100 JavaScript files.  Downloading each of these separately can add a few seconds to the start of Xamin.  So the default version of Xamin uses the features of ExtJS to compress the Java into a single file.  So normally users can access Xamin through either an index.jsp (or no name after the final / in the URL), or xamin.jsp version.  The Xamin.jsp version will download each JavaScript separately, while the default index.jsp version uses the compressed files.

The xaminInstall script requires two arguments: the name of the developer doing the installation (which is used to find a standard location for the input war file), and the type of installation being made.  This determines where the installation is to be done.  A prod or production release goes in the production area of primary HEASARC machines, test goes on HEASARCDEV in the standard location, and dev goes into special areas defined separately for each user.  There are many other options which can be used to override the default behavior. 

Setting up the CLI/Batch interface

Since direct access to the database is not available outside of GSFC the Xamin batch mode gets around this by providing a query proxy to the user. The user need download only a very small executable Jar file that contains two classes. The classes in this jar take the user's input arguments and convert them in to an HTTP POST request to the Xamin web server. If there are file arguments (i.e., UPLOAD) then these files are read and appended to the POST. If the user has specified an OUTPUT argument, then the output is written to the designated file.  The users.jar file is normally accessible as .../xamin/distrib/users.jar.

The server side of the batch interface is a thin layer which initiates a servlet that runs the user specified query and returns the results. The CLI servlet is very similar to the servlet used in processing Web requests except for the handling of user authentication. If a user wishes to access an Xamin account through the batch interface, both the user name password must be supplied in the command arguments. These are authenticated using the HEASARC login service.

To execute the users JAR file the user should enter:

       java -jar user.jar arg1 arg2 ...
        

      java -jar user.jar table=rosmaster,ascamaster offset=a:b:10
        

Code Overview

The Query Library

This sections describes the general flow of control in processing a query with a view to showing how the various classes work together.

Startup

The first order of business for the Controller is to set up three global structures. The Connecter is used to get connections to the underlying database or databases. The Settings are used to manage the system and user settings. The MessageHandler is used to convey information about processing to the user. Each of these is available in a global context. To allow for robust access even in an environment where there may be many simultaneous queries, these three globals use a static ThreadLocal variables to ensure that each instance of a query provides an independent instance to the query processing.

The Connecter allows for reuse of a single connection to the underlying database or when multiple connections are desired to get as many connection as needed. During shutdown it ensures that all connections are closed.

The Settings are essentially is just a Map from keywords to string values. Settings are set by system set up files, user arguments and also by program operations.

The Message Handler provides a standard way to send messages back to the user. Initially it is set to simply echo whatever is sent on the standard error output. Once the query processing has progressed to a stage where the desired output format has been established, a message handler compatible with that format will be used. E.g., for a VOTable the messages will be rendered as INFO elements at appropriate locations in the VOTable.

Help and User account processing

After establishing the Settings the Controller begins to analyze the request the user has made. There are a few non-query requests that are handled first.

If the user has requested help documentation the Help class is used to statisfy the request. If a USER setting has been specified, the the UserAccount class is used to ensure that the password specified is correct (or that the user has consistently forgone password protection on the account).

Next there is a check to see if this is an account processing directive. A user may create an account or modify it using the functions in the AccountManager or view or delete tables using the TableManager.

Setting up the query

After verifying that the request is not one of the non-query requests, the Controller.runQuery method starts setting up the query environment. The first step is determining the output. A StandardDisplay object is created which will handle the communications with the user. A FormatFactory which creates the WriterElements appropriate for the selected format is created and the WriterElement are plugged into the StandardDisplay. WriterElements separate out tasks for writing. Notably there are WriterElements for writing table information and distinct elements (but associated through the FormatFactory) for writing out the data products associated with the element. The global MessageHandler is updated so that messages will be incorporated appropriately into the output, e.g., if the output is FITS, the messages may be rendered as FITS comments.

If the user has requested information about the setup environment, (showArguments or ShowSettings), then these are handled by sending the appropriate messages.

Finding the tables to be queried

The first step in building the query is determining which tables are used in the query. Tables can be either either local (e.g., already present in the database), or uploaded. Uploaded tables will be added to the database temporarily for the duration of the query and then deleted afterwards. If there is a user account associated with the query, then the uploaded table can be persisted.

The Controller looks for a TableFinder class to use. Normally this will be the StandardTableFinder, but in principle this can be overriden by the user.

The TableFinder looks to the TABLE keyword to find any existing tables that are to be used in the query. It creates a QueryBuilder class that will be used to store information about the query and add Tables to it that represent each of the user selected tables. A MetadataInfo object is created that has all of the required metadata for the system by querying the system metadata table. The metadata for tables not participating in the query is not included in the MetadataInfo object.  To maximize efficiency when many Xamin queries are being done, metadata is cached and updated periodically, typically every 5-10 minutes.  Thus when the metadata tables are updated the effect may not be seen immediately.

Next come the uploaded tables. These are specified using the UPLOAD. Format specific classes (TDat, CSV, VOTable or SourceList) implement the Table interface for the input data. A generic TableCopier class then copies this table into underlying database as a temporary table. Metadata for the uploaded tables is added into the MetadataInfo object.

Building the Query

Once the tables that are to used in the query are found, work begins on setting up all of the SQL clauses in the query using a QueryConstructor. User specified constraints are added to the Where clause. There may be explicit constraints using the CONSTRAINT keyword, or are converted to explicit constraints using the user specified time or positional constraints. Constraints use ParsedExpressions which identify the underlying columns used in each constraint. This allows us to deal with the aliases that tables may have in the query. Generally the first table is aliased to 'a' the second to 'b' and so forth. A TableAliaser manages the mapping between tables and aliases.

Positional constraints involve substantial work. User specified positions may be in a variety of coordinate systems and may involve source names. the SkyView SourceCoordinates class is used to handle this. DistanceCalculator classes use different ways to specify a maximum radius constraint for a query. This is used both for point sources and cross-correlations.

The output fields for the query may be defaulted or may be explicitly specified.

As each of the user inputs is processed the QueryBuilder object is updated appropriately. E.g., position, time, and explicit constraints are added in separate steps. Any requested sort order is saved. If the user has requested a grouping clause that can be added.

Running the Query and Security Checking

After all of the user inputs are processed, the QueryBuilder generates the full SQL for the query. The query is validated using the SecurityChecker class.  This uses an SQL parser that we updated from the JSQLParser project.  While this parser does not support all queries available within Postgres it does a pretty good job of general SQL parsing including supporting what is needed for the VO ADQL parsing.  The SecurityChecker class and the parser confirm that the SQL parses successfully, and that it meets our security constraints:  it only accesses table that it should have access to (not
system tables or other users' tables), it only accesses functions on the while list of allowed functions, and that it does not create unexpected artifacts.

Every year or two there is a need to add new functions to the white list of accepted functions.  This requires updating and recompiling the SecurityChecker class.

This SQL is then sent to Posgres with an 'explain' prefix.  This confirms that Postgres can execute the query and returns an very poor estimate of how long the query will take.
 
If there are no problems, then the QueryBuilder returns a Query.  This  is executed to return a Table.

If the user has requested that the results of a query be saved as a user table, then two SQL statements are actually executed. The first statement uses a select into user.xxx ... to run the query and save the result as a user table. The second query is just select * from user.xxx and displays the results of the table that has just been created.

Displaying the results

The Web Client

When the web client starts up, it first queries the Xamin server to get the current list of available tables. It processes any user arguments and then displays the primary Query pane to the user. Users can select tables and and add constraints to queries. Depending upon what the user has done various capabilities are enabled. Eventually the user submits a query. The users inputs are processed, and a query, in the same format a user may use in the CLI interface, is sent using an XMLHTTPRequest to the Xamin server. The results are processed in the web client and displayed to the user in a new pane.

The Xamin Web interface is described in detail in its own document.

Xamin Servlets and JSPs

The Metadata Table

A single metadata table, metainfo, is used to store all metadata for the Xamin system. When a user account is created an extension metainfo table is created in the schema for the user and the join of this user table -- which contains metadata specific to user tables -- and the system table used. The metainfo table uses a generic structure for representing metadata rather than providing a specific structured tables. This allows for great flexibility in the growth of the metadata at the cost of some complexity in the queries and size of the metadata.

To find information in a table we generally need to know the table, row and column we are interested in. Each row of the metadata table is essentially a table, row, and column identifier followed by the information that belongs there.

There are some additional sources of information that are used in Xamin.

Bibliographic information is stored in a the zzbib table but zzbib is not a special metadata table, it is queryable just as any other table. The only special feature of this table is that it is used in the When clauses in a number of conditional products -- but using the same syntax as other conditional links.

Structure of the Metainfo table.

type

The type column is used to specify the kind of data being described. It is analogous to the table name in a conventional representation. The query select distinct type from metainfo is the equivalent of determining what tables are present. Commonly used types include mission, table, product, linkfield, parameter.

name

The name column is used to specify which element of a given type is being looked at. It corresponds to a row identifier in a table view. E.g., if we are providing metadata on tables, then the type field might be 'table' and the name field would be the name of the table. Some care is needed to ensure uniqueness of the name within a type. E.g., it would not enough to simply use the name of the parameter, the name should include the table name and parameter name to ensure uniqueness across all tables.

relation

The relation can be thought of as the column identifier. It describes what the relationship the current piece of information has to the element being described. A table might have a 'description' or an 'author' or any number of characteristics. A parameter may have a 'unit' or an 'ucd'. Some relationships involve other kinds of data. E.g., a table might 'include' a parameter or 'link' to a product.

value

The actual content is presented in this row. E.g., for a table description it will be some text describing the table. For a product template it might be a string that can be used to find files associated with the given product. The value is a text string which in some cases may encode a numeric value.

vorder

Unlike a conventional table, the metainfo structure allows multiple entries with the same type, name and relation. This is often very desirable, e.g., for a table to link to many parameters. In cases where the order of these relationships is important the vorder keyword can be used to specify this ordering. For scalar values or where the order is not significant it can be left null. This integer field is the only non-text field in the metadata.

Current types and relations in the metadata

The Master Tables

Position and Time Tables

Position tables

So to make positional queries with default search radii efficient we separate the tables with relatively small default search radii and index them in the pos_small master tables. Those with large search radii are placed in pos_big. When we want to find all of the tables we run two queries (as a union). The query of the pos_small tables includes a constraint using the maximum default search radius that can take advantage of the dec index. The query of the pos_big tables does not include this additional constraint, but this table is much smaller since only a very few tables have such large error bars.

Time Tables

Position/Time tables

Primary and backup tables

Current views

          select AA from xx_prime
             where table_name not in (select table_name from update_list) and BB
            union
          select AA from xx_bck where BB
        

Indexing

Operations

The scripts are given with respect to xamin/WEB-INF/classes/ops. Logs are generally kept in /www/server/xamin/ops/logs.

Schedule

Acknowledgements and licenses

Included software

ExtJS

pgSphere

nom.tam.fits

com.ice.tar

This work was authored by Timothy Gerard Endres, time@gjt.org.
This work has been placed into the public domain.
You are free to use this work in any way you wish.