The Xamin Guide v1.0

Introduction

What is Xamin?

Xamin can be used through a web interface designed as a Rich Internet Applicaiton (RIA) and also through a command line interface. All facilities are available through both interfaces.

Documentation Overview

The Xamin Guide

This document itself provides a general overview of the Xamin system with particular emphasis on how the system is built and how it can be used. Users interested in an overview of the general capabilities may find this document more accessible than the detailed interface guides. This is the primary document for administrators since it describes in detail the steps for installing and maintaining the Xamin system. Developers may find this document less useful since it discusses the code in only fairly general terms, but the code overview may be a helpful starting point.

This web guide -- designed to be used from the Xamin Web environment -- describes the Web interface in detail.

The Xamin CLI User's Guide

This document describes the use of the Xamin CLI interface. The Web interface is based upon the CLI and the CLI document spends a bit more time on the details and consequences of certain settings, so this document may be useful for both Web and batch users.

The Xamin Plotting User's Guide

This document describes the powerful plotting capabilies. Users may also refer to the STILTS Plotting documentation since Xamin's plot interface is built on top of the STILTS.

The Xamin requirements document.

This document was developed early in the Xamin process to clearly define features required for the final system. The latest version is available on the HEASARC Developmentwiki but requires an account to view. This document is of interest primarily to developers and testers to assure that the implemented systems meets the needs of our community.

The Xamin Design document

This document is of particular interest to developers who may wish to use or extend the Xamin code. It provides details of actual modules used and their relationships.

The Xamin Web User's Guide

Current implementations

    http://heasarc.gsfc.nasa.gov/xamin
        

Migrating from Browse to Xamin

Querying all HEASARC holdings

This works essentially the same as before. Just enter a position and/or time and and start a query. You get back a table with the number of hits for each HEASARC table. Click on the table icon to query a specific table with the specified constraints.

Querying a specific table

In Browse if you wanted to query a specific table you could go to the index page or first select a mission and then the specific table. In Xamin you enter the table in the Tables Explorer quick entry box at the top of the form, or you can browse the the table entries in the Tables Explorer hierarchies. You can also search for tables using a keyword search of their documentation. Any of these can be used to select a table.

Querying all tables from a given mission

This was the default mode for Browse when the user checked a mission box. In Xamin, to select all of the tables for that mission just open the Tables Explorer/mission hierarchy and click on the icon to add all of the tables for that mission. Then use Find matches in... Selected Tables to search all of the tables in the mission for the selected time or positions. You can add tables from other missions, or delete some of the tables before querying if you like. You can then query the selected tables for position or time constraints.

How do I do a query by parameters?

Enter the table name the use the Parameters Explorer to add parameter constraints. You can add a constraint for any parameter by clicking next to it. Note that you can use fairly general SQL expressions in the constraints and these can involve table parameters.

How do I do cross-correlations?

If you have 2-4 tables selected, you can do a correlation of them just by clicking on Query to...Correlate. Spatial cross-match constraints will be added automatically, but you can override or modify these to meet your needs.

Can I search the metadata for tables as I could in the keyword entry interface?

Yes. Use the Tables Explorer keyword search.

How do I monitor a mission for new observations of the target I'm interested in?

Xamin has a broad capability to check if the results for any query have changed. First get a HEASARC services account. Login during an Xamin session and run a query that finds observations you are interested in. Use Query to...Save to save this in your account. Bring up the summary of your saved tables with theSession/User Tables--Info menu. If you click on the Frequency column next to your saved table you should get a pane that allows you to ask to be informed whenever the result of this query change.

System Elements

Interfaces

Web Interface

CLI Interface

Queries

Single Table Queries

Discovery queries

Correlations

All tables can take part in correlations. However remote tables must be downloaded to the HEASARC before the correlation can take place. So in practice correlations of remote resources can only be done if the tables are of modest size or if constraints that can be passed to the remote site can restrict the size to a reasonable number. It may be impractical to query remote tables where the size of the downloaded data is more than ~10,000 rows.

In the web interface spatial correlation constraints are assumed and created for the user -- they can be deleted if not desired. In the command line version the OFFSET keyword is used to specify positional correlations, but the user must include this explicitly.

Temporal cross-correlations can be made as simple constraints on the time columns of the two tables. At the HEASARC times are stored in MJD's, so to ask for observations within a day of one another we can add a constraint of the format

     b.time-1 .. b.time+1
        

     constraint='a.time between b.time-1 and b.time+1'
        

While any number of tables can be correlated in principle, queries tend to become very slow when large numbers of tables are involved. The web interface imposes a maximum of four tables in a correlation, no limit is specified for the command line. This limit can be gotten around if necessary by breaking up the correlation into a series of correlations and saving intermediate results as tables.

Anti-correlations

• A query is performed on all tables of the query including all of the non-group constraints (e.g., all of the constraints that would go in the WHERE clause in the SQL). The fields returned in this query are row identifiers for the non-anticorrelation tables of the query. E.g., if we are querying tables a, b, c, and d with an anticorrelation on table d, then we apply all of the constraints on the query (including constraints on table d) and find out the row tuples for tables a,b and c. These tuples are stored in a temporary table. E.g., one of the results of this query might be that row 5 of table a, row 6 of table b, and row 7 of table c are found, so we save a tuple (5,6,7).
• A second query is now performed where all query constraints that involved table d are removed and a new constraint is added that the any tuple of rows that was found in the first query is excluded. The output fields that the user asked for, and any group constraints (constraints that would be put in a HAVING clause) are added back into the query. The results of this query are returned to the user.

Note that Xamin does not do careful analysis of the input constraints to determine if pieces of an input constraint can be included in the second query. E.g., compare the constraint

     constraint='d.exposure>1000 and c.exposure>1000'
        

     constraint=d.exposure>1000,c.exposure>1000
        

Autocorrelations

When doing an autocorrelation it is usually necessary to make sure that you don't return a row matching itself. A simple way to ensure that is to add a constraint of the form

       constraint='a.id != b.id'
        

Data Products

There are two fundamental types of products:

1. Mission archival products represent the data holdings in the HEASARC archive. These are the mission data from more than two dozen NASA and non-NASA missions. For many Xamin users querying observation catalogs is just a step in finding and locating the datasets of interest to their research. These products and be retrieved and downloaded using Xamin.
2. Link products are URLs that point to data that are related to a specific row, but are not part of the HEASARC archive proper. We distinguish several types of link products.
   • Table links return rows in a linked table that are related to the row in the linking table. E.g., each row of the SwiftMaster table links to the associated rows in the Swift instrument tables (BATMaster, XRTMaster and UVOTMaster). In turn the instrument tables link to the mission master table. Table links can and often do span missions. Table links are provided between many of the major observation master tables, e.g., linking each Chandra observation to nearby XMM observations. Following a table link executes a new Xamin query.
   • Bibliographic links provide links to papers using HEASARC observations. A master bibliography table indicates which observations have been used in which table. The table has been collected manually and does not cover all of the literature for all missions but covers much of the HEASARC�s holdings.
   • Dynamic service links use positional information from each row to query other services. Currently the HEASARC provides service links to NED, SIMBAD and SkyView. Unlike other products these are available for any table without needing to be specifically associated with a table. The mechanism used for positional service links is not restricted to positions. If a remote service can be invoked by specifying a value and the metadata for a table indicate it has a field or fields that have appropriate data, a dynamic link can be made.

Downloading archive data

As data products are collected they are added to a shopping cart. The shopping cart can contain products from multiple selections and from multiple queries. Users can also download individual data products directly (or in the case of directories open a web link to the directory). Link products cannot be added to the shopping cart -- they generally involve data that is intended to be viewed in browser or which are found at remote sites.

At any point in an Xamin session users can download the data in their shopping cart. They can download individual products one by one from the shopping cart, but if there are more than two or three products users may prefer to collect the data products together. A tar file of all of the products in the cart can be created. This tar file is created dynamically so the download should start almost immediately.

Whenever a significant amount of data is to be downloaded we recommend that a data download script be created rather than directly tarring the data. Scripts can be created which use either the WGET or CURL commands both of which are widely available. Scripts are much more easily restarted and allow the users complete control of the data download process. The script can be displayed and copied via striping or it can be downloaded explicitly.

In the command line environment a query can be used to build a download script to download archival products or to provide a list of links to link products.

Setting Up Data Products

• Product descriptions are table independent: they describe the content of a product and where it is located (or how to construct its URLs) in terms of a set of variables. A product may have other products as child products. A child product inherits the variables of its parent. Often products specify a file template which is used to match against the file system relative to a root location. A root variable will point to some directory associated with a specific observation.
• The TableLink metadata sets up a link between a specific table and a product. Many tables can link to the same a product. The table links describe how the values of the variables in the product definitions should be created based upon the columns in the table the product is being linked to. They can also specify a boolean expression that must be satisfied if the product is to exist. This allows for conditional products. E.g., we might wish to link from a table of objects to different linking tables depending upon the classification of the object.

User Accounts

Capabilities

On-line users can also save the current query configuration at any time into a named configuration. This configuration can be restored at any later time. Note that the configuration only affects the Query window, not any of the other Xamin windows. If a configuration is saved with the name default, then auser can automatically restore this configuraition in subsequent logins. To change the default configuration the current default must first be deleted.

Output formats

ASCII

VOTable

VO Protocols

Excel

FITS

Table Sources

Local Tables

User Tables

Uploaded Tables

External Tables

Sample Queries

Discovery Queries.

Examples:

• What HEASARC tables have information near 3c273?
• Does the HEASARC know of any observations made on June 6, 2007?
• What X-ray data does the HEASARC have on Cen-A in 2005-2006?
• What ROSAT, Chandra, XMM and ASCA observations does the HEASARC have on 3c273, Abell 1656 or 10 15 06.2, +14 25 19.8?

Simple Single Table Queries

Examples:

• What does the XMMMaster table look like?
• What observations has Chandra made of 3c273 or 3c279?
• What did XTE observe on 2004-10-10?

Complex Single Table Queries

Examples:

• What observations has ROSAT made of 3c273 that had exposures > 10,000 seconds?
• What are the longest XMM observations?
• What is the average flux of RASSFSC sources as a function of galactic latitude?
• What high-galactic-latitude sources in the XMM Serendipitous Source Catalog have fluxes more than 10 times the flux error?
• For each source observed by ROSAT what were the total number of observations and the total exposure time of all observations of the source per instrument?

Cross-correlations

Examples:

• What Chandra observations have been made of objects in my uploaded source list?
• What targets have been observed by both ROSAT and Chandra?
• What targets have been observed by ROSAT with an exposure of at least 40000 seconds and Chandra with an exposure of 10000 seconds?
• What targets have been observed by Chandra with an exposure time greater than the ROSAT exposure time?
• How many times has each Messier object been observed by Chandra?
• What is the x-ray/radio flux ratio of objects seen in both the XMMSSC and FIRST catalogs?
• Which of the objects in my source list have not yet been observed with Chandra?
• What are the brightest objects classified as AGN in the WGACAT catalog that have not been observed by Chandra?

External Queries

Examples:

• What are 100 nearest USNO-B sources near the north Galactic pole?
• Are there Chandra observations of black holes with masses greater than 100 solar masses? (E.g., J/ApJ/667/131 is a table of black hole masses).

Monitors

Examples:

• Let me know about new XMM or Chandra observations of Abell 496 or Abell 1656.
• Are there any new Chandra ACIS observations with exposures longer than 10,000 seconds?
• Is there any new information for any tables with positional errors less than 1� on TY Pyx?
• Maintain a list of distinct PIs and the number of observations for Suzaku.

Upload Queries

Examples:

Which elements of my source list are at high Galactic latitude?
• Given a CSV file with RA,Dec and Magnitude, find the difference between these magnitudes and magnitudes in the USNO B catalog?
• Given a VOTable with a set of stellar positions, find the objects where the offset with Hipparcos positions is larger than 2� (assuming that any offset >20� is a misidentification).
• Find all the Suzaku observations with exposures > 20,000 seconds but only if there is also a ROSAT PSPC observation of the field.

Installation

This section describes the steps in creating the complete Xamin data access system. The discussion of each step includes both the general issues that need to be resolved by an implementor in their system, and the specific choices made at the HEASARC.

This section describes a complete installation of the Xamin server. Users who only wish to be able to use the Xamin from their CLI environment need only download the very small users.jar file from the Xamin distribution directory (e.g. http://heasarc.gsfc.nasa.gov/xamin/distrib/users.jar).

Installing and populating the database.

Xamin is designed to use a PostgreSQL database. Some PostgreSQL specific features are used in the software to populate the system and in the query system. The keyword search interface uses PostgreSQL features that paralleled in some other systems (e.g., SQLServer), but may not be generally available. Postgres specific features of the system are called out

Postgres is freely available at:

   http://postgresql.org
        

While one click install downloads are available, these generally require superuser privileges on the destination machines..

Installing the database

The tar file should be unpacked with

      > tar zxf postgresql-X.Y.Z.tar.gz

        

Within the postgres-ql-X.Y.Z directory

    > ./configure --prefix=desired_postgres_directory
        

    > ./configure --prefix=/local/db/postgres
        

Then execute:

     make
     make install
        

The current Postgres operational machine is installed on the ASDDB machine.

The Postgres server is ideally run under a special user account but may be run under a normal account. The special account can be created using normal system procedures.

Initial configuration.

To initialize the database first create the directory in which the database data is to be stored. This directory should be owned by the same user as will run the server.

    > mkdir directory_for_data
        

    > initdb -D directory_for data
        

The output from the initdb command indicates how one can start the database server. This command should be put in the startup scripts for your database machine (but is not currently for the HEASARC).

Once the database has been initialized some small amount of customization of the configuration files is desirable. These are found in the data directory mentioned above.

• The pg_hba.conf file should be updated to allow remote access. This can be configured in a number of ways.
• We have also made the following changes to the postgresql.conf file:

     shared_buffers = 500MB
  
     temp_buffers   = 8MB
     max_prepared_transactions = 5
  
     work_mem = 16MB
     maintenance_work_mem = 64MB
     max_stack_depth = 8MB
                  

  All of these lines are updated from the values in the default configuration which is sized for a small machine.

Preparing for data ingest.

Before starting data ingest, an installer will need to download the the needed jar files. Jar files are modified zip files which are used to store Java classes. However they can also be used to store other kinds of files including data files used for running programs. The Xamin jar includes a number of utility programs (mostly written in Perl) that need to be unpacked to be used. The Xamin war file is created in building a full distribution of Xamin. This war file (a jar file containing the complete Web application) is currently created using NetBeans. To unpack the war, copy it to the directory in which you want to unpack it and

   >  jar xf xamin.war
        

Postgres allows for multiple databases to be supported by a single Postgres installation. Each database can have multiple schemas and each schema multiple tables. Cross-database queries are not supported, but cross-schema queries are. All tables used within Xamin should be placed within a single database which may now be created either by using the psql command (the CLI interpreter for the database), or using the createdb command. E.g., using psql we might have:

    > psql
    prompt=# create database xamin;
    CREATE DATABASE
    prompt=#
        

Initializing the database.

The metadata table, metainfo, needs to be created before any other tables can be read into the system. The

    buildmeta.csh
        

    buildfeedback.csh
        

Ingesting HEASARC tables.

The scripts for initializing and populating the HEASARC database are included with the Xamin Web service. The service should be installed as described below. The scripts described below can all be found in the operations area of the installed service (typically something like .../xamin/WEB-INF/classes/ops).

An initial ingest of HEASARC tables may be done using the [run]fullcopy command. The runfullcopy (in ops/ingest) sets up logs and such for the fullcopy command. Just

   > runfullcopy
        

The runfullcopy command uses other scripts in the directory. It calls fullcopy after setting up logs. This calls copyit for each matching TDAT file. The real workhouse inside copyit is pgcopy.pl which parses the TDAT files and creates the SQL to ingest the table. This is then sent to the Postgres psql to be executed.

The procedure populates both the tables themselves and the metadata regarding the tables. However it does not populate the data products related to a table.

In addition to populating the science tables, the runfullcopy command updates the documentation tables within Xamin. Xamin keeps a copy of all HEASARC and Vizier table documentation in the database. These can either be fully refreshed or updated depending upon whether an argument is specified in the command. The updateDocs.pl script is used by runfullcopy to find and ingest documentation.

Creating and updating the documentation table

This resulting SQL file should be run and then the fixdocs.sql script should be run. This last script adds in a bit of HEASARC metadata to the HEASARC descriptions, populates the TSQUERY column in the tabledocs table and gets rid of descriptions of HEASARC tables that are not present in the database.

Ingesting HEASARC product metadata.

The metadata for HEASARC products is ingested using the updateProducts command in ops/products, specifying a directory with the product definition files for each mission. E.g., as the code is distributed if the user makes the directory specified above the default directory then the command

   updateProducts missions
        

The HEASARC has a products file for each mission and special products files for table links, external links, and bibliographic links that should be individually processed.

Building master tables

    buildFullMasters.pl
        

Master tables are views of two underlying tables a _prime table and a _bck. Creating the master tables is fairly expensive because we create a clustered index on fairly large tables. We do not wish to recreate them and regenerate the index each time we update a table. After buildFullMasters.pl completes, the _bck tables are empty. As tables in the database are updated, part of the update process is to add the position/time information from those tables into the corresponding _bck tables. Another table keeps track of the tables that have been updated since the last time we ran buildFullMasters.pl. When a user makes a subsequent query of the master tables, we look in the _bck table for information about science tables that have been updated, and in the _prime table for information about unchanged tables. Since only a small fraction of the tables get updates, the _bck table is much smaller than the _prime table. Thus even though it is not as efficiently indexed, master table queries can run quickly.

Periodically we run buildFullMasters.pl to bring the data from the _bck tables into the primary tables. Since this takes a couple of hours, the approach is to build up the new versions of the master tables using temporary table names. Only when the new master tables are complete do we drop the previous version and rename the new tables to the standard names. This takes seconds.

The heasarc.settings file.

The heasarc.settings file is used to set up the defaults for the Xamin system. The file is normally found in the WEB-INF subdirectory of the Xamin web application. Below we provide an annotated version. Other installations will need to adjust the values to meet their requirements. End users should generally not need to worry about it. It is not used on the client side of the Xamin batch interface.

# The default database
# This will be used as the suffix in the jdbc_xxxx settings
database=postgres

# The JDBC parameters for Postgres

# This is probably the same for everyone.
jdbc_class_postgres=org.postgresql.Driver

# The host and database name will need to be updated.
# The port is the Postgres default port.
jdbc_url_postgres=jdbc:postgresql://db.host:5432/postgres
jdbc_user_postgres=dbuser
jdbc_pwd_postgres=password

# Nominal hosts versus actual host names.  We want to
# be sure to translate any hostname we get into a valid
# host.  So if we get heasarc we want the full path
# heasarc.gsfc.nasa.gov.  If running on a load balancer
# then we want to translate the actual server name
# that we are running on, to the name that we want to
# publish.
heaweb1=heasarc.gsfc.nasa.gov
heaweb2=heasarc.gsfc.nasa.gov
heaweb3=heasarc.gsfc.nasa.gov
heaweb4=heasarc.gsfc.nasa.gov
# What to translate the host "localhost" to.
localhost=heasarc.gsfc.nasa.gov

# These are settings that the user is not allowed to change.
fixedSettings=database,jdbc_class_postgres,jdbc_url_postgres,jdbc_user_postgres

# The User account for non-privileged access
standard_user=dbuser


# Is there forwarding to a TOMCAT server.  This happens only
# for the nominal released and operational versions, not development versions.
# When we are constructing a local URL from the elements available in the
# server environment we may find that the servlet is running on port 6080,
# but we want to publish the original port 80 for the Apache server.
ForwardedPort=80,6080

# The current version of Xamin.
# The first digit is incremented with major releases.
# The second digit is incremented when there is a significant change to user functionality.
# Letters are appended to show bug/documentation fixes that do not appreciably
# change end user functionality.
Version=1.3g

# The following two fields are place holders that we fill in during the installation
# procedure with appropriate versions for the HDB and Xamin projects that are used
# in building the Xamin web application.
HDBVersion=0
XaminVersion=0

# This is replaced by the current date in the installation procedure.
ReleaseDate=2011-01-01


# This is a relative or absolute URL to service that should be invoked
# to request a user login.  Note that Xamin itself does not manage user logins.
# While a user is logged in, all Xamin requests use the login service
# to validate that the indicated user is logged in to the current session.
LoginServlet=/srvlogin

# This is a pointer to the location of the HEASARC hera service.
HeraServer=herasrv.gsfc.nasa.gov
HeraPort=60000
DftHeraName=AnonHera

# These are the E-mail addresses for administrators who will receive notifications
# when Xamin users submit feedback.
feedbackRecipients=user1@xxx.gov,user2@xxx.gov

# These provide a skeleton for the structure of download requests.
wgetcommand="wget -q -nH -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks "
curlcommand="curl -s --remote-name "

# User download requests using the TAR download feature are constrained
# to the tree below.
DownloadPath=/FTP
        

Preparing for VO Access

Preparing the Web server.

Customizing TOMCAT

<Valve className="org.apache.catalina.valves.AccessLogValve"
      directory="logs" prefix="localhost_access_log." suffix=".log"
      pattern="common" resolveHosts="false"/>
        

   <VirtualHost>
        ....
        ProxyPass        /xamin http://localhost:nnnn/xamin
        ProxyPassReverse /xamin http://localhost:nnnn/xamin
        ....
   </VirtualHost>
        

Setting up the Web applications

Xamin is intended to run in a servlet environment and at the HEASARC is developed using TOMCAT. To install Xamin in TOMCAT simply copy the xamin.war file to the TOMCAT webapps directory and restart TOMCAT. This should automatically install the system. Installers may need to update the heasarc.settings file in the webapps/xamin directory.

     mkdir webapps/xamin
     chdir webapps/xamin
     jar   xf some/path/to/xamin.war
        

The Xamin war file contains the full Xamin distribution including operational scripts. Scripts are generally found under the xamin/WEB-INF/ops directory. A site may wish to move these operational scripts to other directories, however the WEB-INF area and its subdirectories are invisible to the outside world so this area can be used without revealing operational details to the public.

Setting up the CLI/Batch interface

Since direct access to the database is not available outside of GSFC the Xamin batch mode gets around this by providing a query proxy to the user. The user need download only a very small executable Jar file that contains two classes. The classes in this jar take the user's input arguments and convert them in to an HTTP POST request to the Xamin web server. If there are file arguments (i.e., UPLOAD) then these files are read and appended to the POST. If the user has specified an OUTPUT argument, then the output is written to the designated file.

The server side of the batch interface is a thin layer which initiates a servlet that runs the user specified query and returns the results. The CLI servlet is very similar to the servlet used in processing Web requests except for the handling of user authentication. If a user wishes to access an Xamin account through the batch interface, both the user name password must be supplied in the command arguments. These are authenticated using the HEASARC login service.

To execute the users JAR file the user should enter:

       java -jar user.jar arg1 arg2 ...
        

      java -jar user.jar table=rosmaster,ascamaster offset=a:b:10
        

Code Overview

The Query Library

This sections describes the general flow of control in processing a query with a view to showing how the various classes work together.

Startup

The first order of business for the Controller is to set up three global structures. The Connecter is used to get connections to the underlying database or databases. The Settings are used to manage the system and user settings. The MessageHandler is used to convey information about processing to the user. Each of these is available in a global context. To allow for robust access even in an environment where there may be many simultaneous queries, these three globals use a static ThreadLocal variables to ensure that each instance of a query provides an independent instance to the query processing.

The Connecter allows for reuse of a single connection to the underlying database or when multiple connections are desired to get as many connection as needed. During shutdown it ensures that all connections are closed.

The Settings are essentially is just a Map from keywords to string values. Settings are set by system set up files, user arguments and also by program operations.

The Message Handler provides a standard way to send messages back to the user. Initially it is set to simply echo whatever is sent on the standard error output. Once the query processing has progressed to a stage where the desired output format has been established, a message handler compatible with that format will be used.

Help and User account processing

After establishing the Settings the Controller begins to analyze the request the user has made. There are a few non-query requests that are handled first.

If the user has requested help documentation the Help class is used to statisfy the request. If a USER setting has been specified, the the UserAccount class is used to ensure that the password specified is correct (or that the user has consistently forgone password protection on the account).

Next there is a check to see if this is an account processing directive. A user may create an account or modify it using the functions in the AccountManager or view or delete tables using the TableManager.

Setting up the query

After verifying that the request is not one of the non-query requests, the Controller.runQuery method starts setting up the query environment. The first step is determining the output. A StandardDisplay object is created which will handle the communications with the user. A FormatFactory which creates the WriterElements appropriate for the selected format is created and the WriterElement are plugged into the StandardDisplay. WriterElements separate out tasks for writing. Notably there are WriterElements for writing table information and distinct elements (but associated through the FormatFactory) for writing out the data products associated with the element. The global MessageHandler is updated so that messages will be incorporated appropriately into the output, e.g., if the output is FITS, the messages may be rendered as FITS comments.

If the user has requested information about the setup environment, (showArguments or ShowSettings), then these are handled by sending the appropriate messages.

Finding the tables to be queried

The first step in building the query is determining which tables are used in the query. Tables can be either either local (e.g., already present in the database), or uploaded. Uploaded tables will be added to the database temporarily for the duration of the query and then deleted afterwards. If there is a user account associated with the query, then the uploaded table can be persisted.

The Controller looks for a TableFinder class to use. Normally this will be the StandardTableFinder, but in principle this can be overriden by the user. This is flexibility is built into many of the key operational classes.

The TableFinder looks to the TABLE keyword to find any existing classes that are to be used in the query. It creates a QueryBuilder class that will be used to store information about the query and add Tables to it that represent each of the user selected tables. A MetadataInfo object is created that has all of the required metadata for the system by querying the system metadata table. The metadata for tables not participating in the query is not included in the MetadataInfo object.

Next come the uploaded tables. These are specified using the UPLOAD. Format specific classes (TDat, CSV, VOTable or SourceList) implement the Table interface for the input data. A generic TableCopier class then copies this table into underlying database as a temporary table. Metadata for the uploaded tables is added into the MetadataInfo object.

Building the Query

Once the tables that are to used in the query are found, work begins on setting up all of the SQL clauses in the query using a QueryConstructor. User specified constraints are added to the Where clause. There may be explicit constraints using the CONSTRAINT keyword, or are converted to explicit constraint using the user specified time or positional constraints. Constraints use ParsedExpressions which identify the underlying columns used in each constraint. This allows us to deal with the aliases that tables may have in the query. Generally the first table is aliased to 'a' the second to 'b' and so forth. A TableAliaser manages the mapping between tables and aliases.

Positional constraints involve substantial work. User specified positions may be in a variety of coordinate systems and may involve source names. the SkyView SourceCoordinates class is used to handle this. DistanceCalculator classes use different ways to specify a maximum radius constraint for a query. This is used both for point sources and cross-correlations.

The output fields for the query may be defaulted or may be explicitly specified.

As each of the user inputs is processed the QueryBuilder object is updated appropriately. E.g., position, time, and explicit constraints are added in separate steps. Any requested sort order is saved. If the user has requested a grouping clause that can be added.

Running the Query

After all of the user inputs are processed, the QueryBuilder generates the full SQL for the query. This SQL is first tested using the for validity and to give an estimate of the time the query will take using Postgres' explain function. If there are no problems, then the QueryBuilder returns a Query object which is executed to return a Table.

If the user has requested that the results of a query be saved as a user table, then two SQL statements are actually executed. The first statement uses a select into user.xxx ... to run the query and save the result as a user table. The second query is just select * from user.xxx and displays the results of the table that has just been created.

Displaying the results

The Web Client

When the web client starts up, it first queries the Xamin server to get the current list of available tables. It processes any user arguments and then displays the primary Query pane to the user. Users can select tables and and add constraints to queries. Depending upon what the user has done various capabilities are enabled. Eventually the user submits a query. The users inputs are processed, and a query, in the same format a user may use in the CLI interface, is sent using an XMLHTTPRequest to the Xamin server. The results are processed in the web client and displayed to the user in a new pane.

The following are the major elements of the JavaScript interface:

xaminStart

Initialize the interface and sets up the framework for the overall window.

xaminQuery

Creates and displays the primary query window. Processes initial arguments to Xamin and extracts the current tables from the Xamin server.

xaminActions

Displays and updates buttons that allow users to initiate queries or clear entered information.

xaminError

Handle errors and messages to be sent to the user.

xaminFeedback

Allow users to send feedback directly to the Xamin system.

xaminGridResults

Display results and data products.

xaminInfo

Display table documentation and other information to the user.

xaminMenu

Render and update menu of options for user.

xaminQI

Process a query request and initiate a query.

xaminSamp

Broadcast results using the VO SAMP protocol.

xaminSession

Initiate and manage user sessions.

xaminShpCart

Manage selected data products and initiate product retrievals.

xaminSpecCon

Support parameter requests and constraints in queries.

xaminStdCon

Support position and time constraints in queries.

xaminTree

Support multiple methods for users to select tables in Xamin.

plotfuncs

Provide support for generating plots in Xamin.

threeaxis

Display and manage plot window.

Xamin Servlets and JSPs

The Metadata Table

A single metadata table, metainfo, is used to store all metadata for the Xamin system. When a user account is created and extension metainfo table is created in the schema for the user and the join of this user table -- which contains metadata specific to user tables -- and the system table used. The metainfo table uses a generic structure for representing metadata rather than providing a specific structured tables. This allows for great flexibility in the growth of the metadata at the cost of some complexity in the queries and size of the metadata.

To find information in a table we generally need to know the table, row and column we are interested in. Each row of the metadata table is essentially a table, row, and column identifier followed by the information that belongs there.

There are some additional sources of information that are used in Xamin.

Bibliographic information is stored in a the zzbib table but zzbib is not a special metadata table, it is queryable just as any other table. The only special feature of this table is that it is used in the When clauses in a number of conditional products -- but using the same syntax as other conditional links.

Structure of the Metainfo table.

type

The type column is used to specify the kind of data being described. It is analogous to the table name in a conventional representation. The query select distinct type from metainfo is the equivalent of determining what tables are present. Commonly used types include mission, table, product, linkfield, parameter.

name

The name column is used to specify which element of a given type is being looked at. It corresponds to a row identifier in a table view. E.g., if we are providing metadata on tables, then the type field might be 'table' and the name field would be the name of the table. Some care is needed to ensure uniqueness of the name within a type. E.g., it would not enough to simply use the name of the parameter, the name should include the table name and parameter name to ensure uniqueness within the entire system.

relation

The relation can be thought of as the column identifier. It describes what the relationship the current piece of information has to the element being described. A table might have a 'description' or an 'author' or any number of characteristics. A parameter may have a 'unit' or an 'ucd'. Some relationships involve other kinds of data. E.g., a table might 'include' a parameter or 'link' to a product.

value

The actual content is presented in this row. E.g., for a table description it will be some text describing the table. For a product template it might be a string that can be used to find files associated with the given product. The value is a text string which in some cases may encode a numeric value.

vorder

Unlike a conventional table, the metainfo structure allows multiple entries with the same type, name and relation. This is often very desirable, e.g., for a table to link to many parameters. In cases where the order of these relationships is important the vorder keyword can be used to specify this ordering. For scalar values or where the order is not significant it can be left null. This integer field is the only non-text field in the metadata.

Current types and relations in the metadata

The Master Tables

Position and Time Tables

Position tables

So to make positional queries with default search radii efficient we separate the tables with relatively small default search radii and index them in the pos_small master tables. Those with large search radii are placed in pos_big. When we want to find all of the tables we run two queries (as a union). The query of the pos_small tables includes a constraint using the maximum default search radius that can take advantage of the dec index. The query of the pos_big tables does not include this additional constraint, but this table is much smaller since only a very few tables have such large error bars.

Time Tables

Position/Time tables

Primary and backup tables

Current views

          select AA from xx_prime
             where table_name not in (select table_name from update_list) and BB
            union
          select AA from xx_bck where BB
        

Indexing

Operations

The scripts are given with respect to xamin/WEB-INF/classes/ops. Logs are generally kept in /www/server/xamin/ops/logs.

Schedule

Acknowledgements and licenses

Included software

STILTS

ExtJS

pgSphere

nom.tam.fits

com.ice.tar

This work was authored by Timothy Gerard Endres, time@gjt.org.
This work has been placed into the public domain.
You are free to use this work in any way you wish.