Xamin Command Line Users Guide

The Xamin catalog and archive interface allows users to query HEASARC catalogs, plot results and download associated data products. This document describes how to use the command line interface to Xamin.

Installation

The Xamin command line interface is distributed as a very small Java executable Jar file. Users will need a version of the Java runtime environment (JRE, version 1.6 or higher is recommended), to run this file. If a JRE is installed, then on Windows machines (and perhaps other environments), the file can be executed directly. In Unix the Jar is run using the command.
   java -jar jarname.jar command-line-arguments
Many shells have facilities available to assign a more natural name to the command. E.g., many Linux shells have an alias command like
   alias runquery='java -jar jarname.jar'
After setting the alias the user can simply execute
   runquery command-line-arguments
The exact syntax may differ on your system.

To use Xamin, you will need to download the users.jar file. This file is available in Xamin's distribution directory. E.g., if the Web interface is at

http://host.edu/xamin
you can download the user jar file from
http://host.edu/xamin/distrib/users.jar

The HEASARC provides a version of users.jar . This can be adapted for use at other sites. By default the users.jar expects to query the HEASARC database. If you wish to access an Xamin service somewhere other than the standard address, then the XAMIN_CLI_URL environment variable should be set to location of the URL that receives Xamin CLI requests. By default this is http://heasarc.gsfc.nasa.gov/xamin/cli. If a test version of Xamin were installed at http://test.mysite.edu:8080/mydb, then the URL would be http://test.mysite.edu:8080/mydb/cli.

Settings

The command line interface to Xamin uses the concept of settings. Settings are just a mapping from case insensitive string keys to string values. In some cases a settings may have multiple values. A user can specify settings on the command line with
     runquery key1=value1 key2=value2
The syntax
    runquery key1=value1a key1=value1b key2=value2
is equivalent to
    runquery key1=value1a,value1b key2=value2
I.e., you can specified multiple values for setting either by repeating the key or using commas in the settings string. If values include spaces or other characters that may be interpreted by the command shell, then the user should escape them using whatever techniques are appropriate, e.g., enclosing the value within quotes. In some cases if a user wishes to include a comma within a settings value if can be escaped using a backslash. Note that some care may be needed in ensuring that the backslash is sent to command and not swallowed or interpreted by the shell. Similarly sometimes you need to ensure that quotes are actually seen by the Xamin CLI processor. For some shells a syntax like runquery key=\'quotedvalue\' is then appropriate, and we use that syntax in examples below.

Simple queries

To run a query try
    runquery table=messier
This will do a query of the entire Messier catalog using the default format and output fields. The results might look like
name |alt_name  |ra     |dec   |constell|dimension|vmag|vmag_uncert|class
M 7  |NGC 6475  |17 53.9|-34 49|SCO     |    80   | 3.3|null       | 3600
M 69 |NGC 6637  |18 31.4|-32 21|SGR     |     7   | 7.7|null       | 3080
M 70 |NGC 6681  |18 43.2|-32 18|SGR     |     8   | 8.1|null       | 3080
... 103 lines deleted ...
M 52 |NGC 7654  |23 24.2| 61 35|CAS     |    13   | 6.9|null       | 3600
M 81 |NGC 3031  |09 55.6| 69 04|UMA     | 26X14   | 6.8|null       | 6200
M 82 |NGC 3034  |09 55.8| 69 41|UMA     |  11X5   | 8.4|null       | 6600

Number of rows:    109
Number of columns: 9

A very common query is to ask whether a table has information about a given position. You can make that query as

    runquery table=rosmaster position=3c273
or
    runquery table=rosmaster position=187.25,2.05
or
    runquery table=rosmaster position='12 29 06.70,02 03 08.6'
In the last example the quotes are included for the local shell and are not seen by the command. Conventionally a comma can be used to separate the ra and dec (or lat and lon) fields of the coordinates though it is not always required.

You can ask for more than one position in a query.

    runquery table=rosmaster position='3c273;3c279;abell 1656'

We may wish to put constraints on the rows we get out. E.g., if we want to skip short ROSAT exposures we might try:

    runquery table=rosmaster position=3c273 radius=10 constraint=exposure>10000

Complex Queries

While simple queries involve only one table, many queries combine information from multiple tables. Queries may involve correlations (joins in database parlance) where each row in the output includes columns from rows in different tables that have been matched in some way. E.g., we might perform a positional correlation between two tables looking for those targets that have been observed by two different observatories. Each row of the result comes from the same set of tables. To specify a correlation you just specify multiple tables in the query and indicate how the tables are to be joined. E.g.,
    runquery table=rosmaster,ascamaster offset=a:b:2
This query looks for ROSAT observations where there is a nearby ASCA observation (the ASCA center is within 2' of the ROSAT center). The 'a' and 'b' aliases refer to the tables in the query in the order they were specified (but see below for uploads).

We can add a constraint to a correlation too.

   runquery table=rosmaster,ascamaster constraint='a.exposure>b.sis_exposure' offset=a:b:30
Or we can ask for a looser correlation but require that the ROSAT observation be longer than the ASCA SIS observation. By default correlations only return data from the first table in the correlation. If we want to show fields from both tables, we might have:
   runquery table=rosmaster,ascamaster constraint='a.exposure>b.sis_exposure' \
      fields=a.standard,b.sequence_number
The FIELDS setting lets us request specified fields (or expressions) for output.

There are other ways to build queries that involve more than one table.

In a union each row is from a single table, but different rows may come from different tables. Basically we run the same query in different tables in succession. E.g., we might want to know all targets that have been observed by any of a set of telescopes. Unions require that the fields being used in the union be identical among the tables in the union. To find all of the ASCA and Chandra observations made of a given Messier object we could try:

          runquery table=messier,ascamaster uniontable=ascamaster union=chanmaster \
            fields=a.name,b.ra,b.dec offset=a:b:30
      
This query first matches the Messier objects against the ascamaster table, and then against chanmaster. Note that if any fields from the table that is replaced in the union are used as constraints or display fields, they must have the same name and type in all of the tables used in the union. E.g., we cannot display the target name in this union because it is called target_name in ascamaster and just target in chanmaster.

Anticorrelations are a special case of correlations. Here rather than finding pairs of tables that match we look for rows in the primary table (or tables) that do not match any row in the anticorrelate. Since there is no matching row, we cannot include any information from the anticorrelating table in the output fields. To find the Messier objects that have not been observed by XMM we might try:

         runquery table=messier,xmmmaster offset=a:b:30 anticorr=xmmmaster
      
When doing an anticorrelation no fields from the anticorrelating table may be used in the FIELDS setting -- how could one display information from the table we don't have information from! -- but fields from that table may be used in constraints. E.g., we might want to find Messier objects that have not been seen in long XMM observations -- we don't care if they have been seen in short ones. To find those we could add constraint=b.duration>20000 to the previous query. It may be counterintuitive, but adding this constraint actually increases the number of rows returned, since we are now anticorrelating with only a portion of the xmmmaster catalog.

Another special class of correlations are autocorrelations, where we match a table against itself. E.g., if we have a table of the many observations of many sources, then we might ask which sources have observations which show large changes in the flux. We might constrain the result to find pairs of rows where the target id is the same, but the magnitudes differs by more than some large value. Here's another simple example. Let's find pairs of nearby Messier objects:

          runquery table=messier,messier offset=a:b:30 \
              fields=a.name,b.name showoffsets constraint='a.name < b.name'
      
The offset looks for pairs that are within 30' which the constraint ensures that we don't return a row matching itself. Note that by using a.name < b.name rather than a.name != b.name in the constraint we also ensure that we don't get two matches for each pair with the order of the match just reversed. We add in SHOWOFFSETS, so that we can see how far apart the pairs are.

Xamin supports all of these kinds of queries and allows various combinations. Very complex queries may need to be built up using intermediate tables to store results. The table below suggests the keywords that are needed to build complex queries. In the Query column we give a symbolic representation of the query for nominal tables A,B,C,D. Here '|' indicates a union, 'x' indicates a correlation and '^' indicates an anticorrelation. In the options where correlations are involved, we've generally included example offset keywords, but it is possible to do correlations between tables in a number of different ways so these should not be considered definitive.

DescriptionQueryOptions
Single table Atable=A
Correlation AxBtable=A,B offset=A:B:size (or other constraint)
AnticorrelationA^Btable=A,B offset=A:B:size anticorr=B
AutocorrelationAxAtable=A,A constraint=...
Anticorrelation of correlated tables (AxB)^C table=A,B,C anticorr=C offset=A:B:size1,A:C:size2
Simple union A|B|Ctable=A union=B,C
Correlation with union (A|B|C)xDxE table=A,D,E offset=A:D:size1,A:E:size2 uniontable=A union=B,C
Anticorrelation against union A^(B|C|D) table=A,B uniontable=B union=C,D anticorr=B offset=A:B:size
Union of anticorrelations(A^B)|(A^C)|(A^D)table=A,B uniontable=B union=C,D anticorr=B offset=A:B:size showanti
Anticorrelation of union (A|B|C)^D table=A,D offset=A:D:size union=B,C uniontable=B anticorr=D
Correlation with a union followed by anticorrelation (Ax(B|C))^D Run as two queries. E = Ax(B|C), then E^A

Tables and Table Uploads

Users can query one or more tables in a given query. If multiple tables are given, then the query is of the join between the two tables. Almost always that involves some kind of correlation between the two tables often, though not always a spatial join. There are three two settings that define tables in a query: TABLE and UPLOAD. The TABLE setting is used to reference a table that is already in the database. UPLOAD is used to specify that a file in some supported format is to be uploaded temporarily into the database. Unless a SAVEAS setting is specified, the uploaded table will be deleted immediately after the query finishes.

Currently uploads can be simple source lists (with one position per row), CSV files, VOTables or the HEASARC's own TDAT format. The upload looks at the first few characters to determine the type or it can be specified by the file type.

E.g., suppose the file mysources.txt has the content:

    3c273
    abell 1656
    10 10 10, -20 24
    12.98, 14.87
We can run a query
    runquery table=rosmaster upload=mysources.txt offset=a:b:30
which will look at the rosmaster table for all observations within 30 arcminutes of the four positions specified in the mysources.txt file. Note the OFFSET setting which indicates that the offset between the 'a' table and the 'b' table should be less than 30 arcminutes. The 'a'-'z' prefixes or aliases are assigned in order first to any tables specified in the query and then to any uploads. E.g., suppose you have a VOTable of a set of interesting Spitzer observations in spizter.xml, and you also have a list of sources you are interested in in mysources.txt. You can ask if any of your sources have been seen by all of ROSAT, Chandra and Spitzer using
   runquery table=rosmaster,chanmaster \
        upload=spitzer.xml,mysources.txt
	offset=a:b:10,a:c:10,a:d:30
Here table a is rosmaster, b is chanmaster, c is the table uploaded in spitzer.xml and d is your source list. You can think of this command as first finding all of the ROSAT observations that are within 30' of your sources and then seeing whether there are Chandra and Spitzer observations near the ROSAT observation. If the OFFSET had be specified as
    offset=d:b:10,d:c:10,d:a:30
the results might be slightly different. Here the positional matching is all done to the source list rather than the ROSAT observations.

Often you will want to upload a table and save the results for future queries. that is easy if you have an Xamin account.

User accounts

The command line interface supports Xamin user accounts. If the USER and PASSWORD settings are specified then user tables can be queried or created. The initial setup of a user account is best done using the Web interface since account validation usually requires responding to a mail message.

If you have an Xamin user account, then you can access the tables from that account exactly like system tables except that you prefix their names with the account name.

   runquery user=xxx password=yyy table=chanmaster,xxx.savedquery offset=a:b:10
runs a cross-correlation between the chanmaster table and a table that the user had saved earlier. The MYTABLES setting can be used to view user tables and the DELETE setting can be used to delete a user table. E.g.,
   runquery user=xxx password=yyy mytables
will return a list of your tables.
   runquery user=xxx password=yyy delete=atable
will delete the table xxx.atable

Saving uploads and query results

By default a table (or source list) is uploaded as a temporary table and is destroyed as soon as the current query finishes. If you wish to save an uploaded table then specify the saveas setting as part of the upload. If you do not wish to view the table as part of the upload you can also set format=none to suppress the display of the uploaded table. E.g.,
    runquery upload=mytable.xml
will display the contents of mytable.xml but not save it. If this table has a mag column you might wish to filter it
   runquery  upload=mytable.xml constraint='mag>10'
to see only the fainter rows.

If you want to save the table you are uploading for future use then

   runquery user=xxx password=yyy upload=mytable.xml saveAs=bigstars system
will display the upload and save it. The SYSTEM setting ensures that certain system columns that are likely to be useful in future queries are included. If you already know what the table looks like you can suppress the output with
   runquey user=xxx password=yyy upload=mytable.xml saveAs=bigstars system format=none

Note that the saveAs command saves whatever the query results would be. It saves the filtered table and only the fields in the table you specify.

It is an error to attempt to overwrite a table using SAVEAS. Use the DELETE setting first.

Any query can be saved using SAVEAS. Sometimes when doing complex queries it can be faster to break the query into pieces and save the intermediate results. If you have some filtered version of a table -- or perhaps some correlation result -- you can save that as your own table.

To save a personal version of only the northern Messier objects we might try

query user=xxx password=yyy table=messier constraint='dec>0' saveas=northmessier system
which will create the table xxx.northmessier.

You can specify an existing table with with saveas. Xamin will attempt to append the results of the current query to the existing table. The fields being retrieved must be the same. E.g., you might have a very large table that times out when you try to upload it. Break it up into pieces and upload it in sections.

Query constraints

A typical query involves constraints on the table or tables being queried: only rows that meet specific criteria should be returned. This section discusses how constraints may be placed on queries. There is special support for positional and temporal queries since these are so common. Generic constraints can be specified on any column in the table. Users can also build complex queries using SQL aggregate functions.

Positional Constraints

Positional constraints can be specified using the POSITION and RADIUS settings or for multiple table queries using the OFFSET setting. The user can specify one or more positions in the sky with POSITION: only rows where the primary position is within the specified RADIUS of any of the positions are returned.

If a single position was specified, then the SHOWOFFSETS setting can be used to include an offset column between the specified position and the position in the current table. This offset column is not available when multiple positions are specified: it would be unclear which of the input targets the offset would refer to. If you need to get offsets from multiple positions, upload a SOURCES list and do a cross-correlation with that resulting table.

If multiple tables have been specified in the input, then the POSITION constraint applies to the first table that has position fields. If no RADIUS setting is given, then a table-dependent default is used. For observation tables the default is a typical size of the observation. For object catalogs the default may be the uncertainty in the position.

The radius constraint can be specified as a expression and not just a constant. E.g., the table might have a column, pos_err giving the positional error of the each row. One could then specify

    position=3c273 radius=2*pos_err
to ask for any objects within twice the position error of 3c273.

If an expression is used for the RADIUS setting it can sometimes be desirable to add a second constant RADIUS constraint that provides some maximum value. The can significantly improve query performance. E.g., if the maximum position error in our table is 30", then we might specify

   position=3c273 radius=2*pos_err,1
The second value for the radius indicates that we must be within a 1' (i.e., 60") of the source. Multiple radius settings are often useful when querying the position master tables.

Positional cross-correlations

When multiple tables are being queried, users normally wish to do a positional cross-correlation between the tables. The OFFSET keyword is used to specify this constraint. Each OFFSET specifies a constraint between two tables. The offset is specified as three tokens
    alias1:alias2:max_sep
where the first two fields give the aliases for the two tables on which the constraint is being applied, and the third gives the maximum separation allowed between the two tables. This third field can be default to the table-dependent default search radius where the first table's default is used. If more than two tables are being used in the query then multiple offset fields may be used. E.g.,
    runquery table=rosmaster,chanmaster,ascamaster \
       offset=a:b:10,a:c:30
requests data where there are concident ROSAT, Chandra and ASCA observations where the center of the Chandra observation is within 10' of the ROSAT observation and the ASCA observation is within 30' of the ROSAT observation. An offset can pair any two tables in the query (so long as the table has positional information).

When you make an N-table query, the program may give you a warning if it does not see at least N-1 offset values. This can mean that you forgot to constrain one of the tables involved. You can ignore this warning if you understand the results you are getting. E.g., you may have joined the tables using a non-positional constraint.

If the SHOWOFFSETS setting is specified, then a column indicating the offset between the two tables is included for each offset constraint requested. The name of the column gives the aliases of the tables involved.

Temporal Constraints

HEASARC tables store times using integer and floating point values using Modified Julian Days (MJD). Users can query columns directly but the TIME and TIMEDELTA settings provide some special support for temporal queries. Users can specify time instants and ranges using the TIME setting in either Julian Days, MJD or ISO formats. Generic constraints on temporal columns can only use MJD. Also, the names of the columns used to store times differs among tables. Using the TIME setting, the program examines the table's metadata and finds the start and stop time fields and specifies the proper query.

When users specify a specific time rather than a time range, then this time is normally extended into a range by adding a TIMEDELTA tolerance which defaults to 1 day. To query for coverage of an instant, you also specify TIMEDELTA=0.

Time ranges are generally specified using the .. operator. E.g., so include all of 2003 one might specify

    time=2003-1-1..2004-1-1

The input format used for the ISO style format is officially

    yyyy-mm-ddThh:mm:ss.ffff
but the parsing used is fairly forgiving. Note that the date specified without a time represents the beginning of the day. To query for a given day a user might try
    time=2003-10-17..2003-10-18
or alternatively
    time=2003-10-17T12:00 timedelta=0.5
The first specifies a range explicitly, while the second species an instant in the middle of the day and uses the TIMEDELTA keyword to expand that instant to the full day.

Temporal cross-correlations

Currently there is no special support for temporal cross-correlations but they can easily be made using generic constraints (see below). E.g., if we want to find overlapping ROSAT and ASCA observations that started within a day of one another:
   runquery table=rosmaster,ascamaster ...
      constraint=abs(a.start_time-b.time)<1
where we might expect that there are some other constraints on the query as well. Note that the start time fields for the two tables differ.

Generic Constraints

The CONSTRAINT setting is used to add any constraint to the query the can be expressed as an SQL logical expression. Any number of constraints can be placed on the query. Constraints can refer to any number of tables within the query -- and even to to tables not directly in the query.

A simple contraint might be no more than

    constraint='bmag>10'
[where the quotes are to escape the > in the expression] asks for fainter objects. If the user wanted a range of magnitudes
    constraint='bmag between 10 and 12'
A range operator .. can also be used as a shorthand here.
    constraint='bmag=10 .. 12'

If this is a correlation we might want to find only those rows where the magnitude appears to change between two tables then

    constraint='abs(a.bmag-b.bmag) > 1'
This assumes that both tables have bmag column.

Some special handling is allowed for string matches. If a constraint is given as

    constraint=col='xxx'
and the string xxx contains an '*', then this is assumed to be a wildcard search. It translates to the SQL
    col like 'xxx'
rather than col='xxx'.

When doing string queries CLI users usually need to specify the quotes to designate string constants. The constraint

    constraint=pi_lname=white
looks for rows where the column pi_lname has the same value as the column white. The constraint probably should be specified as
    constraint=pi_lname=\'white\'
The syntax given is appropriate for many Linux shells and makes sure that the Xamin CLI sees the string pi_name='white' -- i.e., the quotes are not processed by the shell.

Aggregate Based Constraints

The Xamin CLI allows the use of the GROUP setting to aid users who wish to use SQL aggregate functions (e.g., avg, count, stddev, min, max) functions in doing analysis of a table. If the user wishes to apply a constraint that involves one of these aggregate functions then it should be specified in the CONSTRAINT setting as with other non-aggregate constraint. E.g., suppose we want to find all of the Messier objects that have been seen by Chandra at least 10 times. We might have
   runquery table=messier,chandra fields=a.name,'count(*)' \
         group=a.name constraint='count(*)>10' \
	 offset=a:b:10
When an aggregate function is found, the constraint is added to the HAVING clause of the SQL query rather than the WHERE clause where ordinary constraints are applied.

Special Tables

There are several special tables in the HEASARC database that are useful for specific kinds of queries. The master tables can be used for discovery queries that determine which tables might have information of interest. The metadata table supports all of the HEASARC's metadata.

Query the master tables

The master tables are simple tables that store the time and position information from each of the HEASARC's constituent tables plus a link back to the original table. The only other column they have is an indication of the default search radius appropriate for that row (almost always a function of the original table). One can very quickly query the master tables to find out which standard tables have information at a given location or time. E.g., to find out all the tables with information on 3c273
    query table=pos_small position=3c273 fields=table_name,'count(*)' \
       group=table_name radius=dsr,60
will do the trick.

To speed up access, the master tables are broken up into two groups. The pos_small and pos_time_small table includes all of the information in the HEASARC where the default search radius equal to or less than a degree, while pos_big and pos_time_big includes those with default search radii that are greater. The very large default search radii of these few tables significantly slow down queries where we are looking to see if a given position in within the default search radius of a row. The ill-defined positions of the data in pos_big is also rarely of interest to users looking for data in specific locations. So typically users will only wish to query the pos_small table.

If only temporal queries are being made, then the time table can be used. E.g.,

    query table=master_table.time time=1990-10-10..1990-10-12 \
       fields=table_name,'count(*)' \
       group=table_name
There are a total of five master tables. These are all in the master_table schema and must be referenced as master_table.xxx when queried.
pos_small All rows that have positional information with a positional error less than 1 degree.
pos_time_small All rows that have both positional and temporal information with a positional error less than 1 degree.
pos_big All rows that have positional information with a position error greater than 1 degree.
pos_time_big All rows that have position and temporal information with a position error greater than 1 degree.
time All rows with temporal information. No positional information is included in the table
Generally for queries of position alone, the pos_(small|big) tables should be used. For queries involving position and time pos_time_(small|big) tables can be used. These are faster, since they have many fewer rows. The time table should be used for time-only queries.

Querying Metadata

The metadata for Xamin is stored in the table metainfo. If a user account is being used, then the the table user.metaview is a view that includes both the user and system metadata. These tables can queried like any other. A few useful queries are suggested here:

What tables are in the system?

   runquery table=metainfo distinct fields=name sortvar=name constraint=type=\'table\'
What are the parameters for the table xxx?
   runquery table=metainfo distinct fields=value sortvar=value
      constraint=type=\'table\',name=\'xxx\',relation=\'contains\'
What tables do we have that include information in the radio regime?
    runquery table=metainfo fields=name sortvar=name \
       constraint=type=\'table\',relation=\'regime\',value=\'Radio\'
What fields in ROSMASTER have a unit of mjd?
    runquery table=metainfo constraint=type=\'parameter\' \
       constraint=name=\'rosmaster.*\' \
       constraint=relation=\'unit\' \
       constraint=value='\mjd\' \
       fields=name
In this last query we split up the constraints to show them separately rather than putting all four constraints into a single setting.

Output Columns

The columns included in the output are controlled by the three settings FIELDS, SHOWOFFSETS and SYSTEM. The FIELDS setting can be used to request that a standard set of columns or all columns for a given table be included, or it can specify an explicit list of columns or expressions. E.g,
    fields=name,ra,dec,'vmag-bmag as color'
asks for four columns. The last is a difference of two columns and we can use the standard SQL syntax to give that column a name -- though that is not required.
    fields=name,ra,dec,vmag-bmag
would work fine too, but would use whatever the database comes up with (perhaps a blank) as the column title.

The SHOWOFFSETS setting requests that where possible the system add in a column giving the offset between the table row and the requested position or between two table rows. There can be multiple offset columns in a given query when doing correlations.

Internally Xamin tables have some special columns added to them. An __row is a simple integer index of the table. If the table has positional columns than a set of unit vector columns is created with names like __x_ra_dec, __y_ra_dec, __z_ra_dec. The initial x,y,z indicates the element of the spatial unit vector, while the ra and dec indicate the coordinate columns these unit vector columns relate to. Currently unit vectors are only created for RA and Dec columns.

The position columns are used for positional queries, so that such queries will not be possible if these queries are deleted. When copying a table result using the SAVEAS, specifying SYSTEM ensures that these system columns are also included.

Output Formats

The FORMAT setting is used to control the output format used in displaying the data. The default aligned format is a simple ASCII format where columns are padded to be the same size for all rows and pipes ('|') are used to delimit columns. A stream format foregoes the padding which means that results can be written out immediately as they are read from the database.

An EXCEL format provids a simple spreadsheet result. It uses the .xls style files rather than the new Excel XML formats.

FITS data can be written out a as a binary table. A dummy header HDU is prepended to make the output a valid FITS file.

The VOTable XML format is fully supported. If this format is selected an XML XSLT stylesheet can be used for further processing. the FILTER keyword should give the name of the XSLT file.

Products

Some of these output formats support display of data products. The PRODUCTS setting is used to indicate that display of products is desired. It can be given without a value in which case the base products for the tables being queried are displayed, or a specific set of products can be chosen. Products are displayed in format specific fashion. Currently products are supported in the aligned, stream and VOTable formats.

In the VOTable formats, products are added to the rows of the output using an extension to the VOTable format. This extension is not recognized outside the HEASARC and such VOTables should not be expected to work in external software. An XSLT stylesheet can trivially convert these formats back to a valid XML. (Simply delete <Products> and their subnodes).

Settings Summary

Table sources
Table Local database table[s] for queries or correlation.
Upload VOTable[s] to query or correlate (also UploadData)
Sources List[s] of sources to use for positional queries (also SourcesData)
Union List of tables to query sequentially in union
UnionTable Table to be replaced by union tables
AntiCorr Table to be anticorrelated against
Divider Divider string for CSV/TSV/... style files.
HeaderLines Number of header lines in CSV/TSV/... style files.
Query control
Position One or more targets for cone search
Radius Radius for position or sources match (arcminutes)
Time Specify the time to be used in the query
TimeDelta Look for the given interval around an instant.
Coordinates Coordinate system to be used [also icoordinates and ocoordinates]
Offset Positional offset constraint between tables
Distinct Do not show repeated rows
Group Group outputs
Constraint SQL contraint to be satisfied by output rows
Sortvar Fields/expressions to sort output by
SkipQuery Don't run the query (with showquery or showcost)
TimeLimit Only run queries with shorter guesstimated times
Resolver Use the given resolver[s] to convert target names to coordinates
EndTimeOptional Allow endtimes to sometimes be null
OuterJoin Support an outer join on a table
Output control
Format The format of the result (votable|stream|aligned|Excel|FITS)
Saveas Save a table into the user's space
ShowOffsets Include offset columns in results
ResultMax Maximum number of rows to display
Fields Rows or expressions to be displayed
Products Include data products in the output
System Include system fields in the output
ShowAnti Show anticorrelating table name[s] in results
Generic Allow generic data products
Metadata
ShowQuery Show the SQL query that will be sent to the database.
ShowCost Show a guesstimate on how long the query will take
TimeLimit Forgo/stop query if estimated/actual time exceeds limit.
User
User The name of the user account
Password The password for the user account (not required)
Account Manage the account
MyTables List the tables in the current account
Delete Delete a table from the current account

Settings Details

You can get help information in the batch interface itself by doing
    query help
or
    query help=subject
where subject is a particular topic or command. E.g., to get help on the table setting, just enter
    query help=table
The keyword is case-insensitive for all settings. Keyword values may be case sensitive depending on their context.

AntiCorr

The name (or alias) of a table against which an anticorrelation is to be done. If an anticorrelation is desired against a correlation, then the correlation should first be performed save. No fields from the anticorrelating table may be requested as output fields.

The anticorrelation is done in two stages. In the first step the query is performed as a correlation, and all sets of matching rows in the tables other than the anticorrelating table are determined and saved. Then the query is re-executed and the anticorrelating table and any constraints that involve the anticorrelating table are removed. An additional constraint is added such that any row that was returned in the first stage is not included. Constraints that do not involve the anticorrelating table are included in both queries.

E.g., suppose we want to find all of the targets seen by both ROSAT and ASCA whose ROSAT observation time is longer than the ASCA SIS observation time. However we are not interested in those targets that have already been observed by Chandra with an exposure greater than 1000 seconds. This query might be set up as

    table=rosmaster,ascamaster,chanmaster
    offset=a:b:10,a:c:10
    anticorr=chanmaster
    constraint=a.exposure>b.sis_exposure
    constraint=c.exposure>1000
In the initial query we find all of the nearby pairs of ROSAT and ASCA observations meeting our exposure constraints and which are near a Chandra observation with a long exposure. Next we run a correlation on just rosmaster and ascamaster where we keep the constraint on the ROSAT and ASCA exposures, but we drop any matches we found in the first query.

Example

 tables=messier,ascamaster offset=a:b:10 anticorr=ascamaster
asks for the tables in the Messier catalog for which there is no ASCA observation within 10'.

Constraint

An expression specifying a constraint that you wish to be satisfied on each output row. This can be any logical valued SQL expression.

Examples

   constraint='vmag<4'
   constraint='a.vmag>b.vmag+2'
   constraint='a.ra<3 or a.ra between 7 and 8'
Note that the quotes are not part of the argument. They are included to escape the shell processing of special characters like < or space.

Coordinates [ICoordinates,OCordinates]

The coordinate system used for position input and ouput. It can be overriden for either using the icoordinates or ocoordinates setting The coordinates should be a string of the format
(J|B|E)nnnn or Galactic or ICRS.
Here J indicates Julian coordinate, B, Besselian coordinates, and E, Julian Ecliptic coordinates. The nnnn is a numeric epoch.

Examples:

   coordinates=J2000
   coordinates=Galactic
   icoordinates=Galactic,ocoordinates=E2000

Delete

Delete a user table. The user and password settings must also be specified. Note that the name of the table to be deleted does not include the user name.

Example:

    user=someuser delete=myfirsttable

Distinct

Show only rows that differ. Since not all columns of a table or join are shown it is possible that two rows can be output with identical values. Setting this keyword suppresses such rows. This can be particularly useful if you are doing a cross-correlation, and want to get only the rows in one of the tables.

E.g.,

   query table=chanmaster,rosmaster offset=a:b:5
will show more than 30,000 results repeating many rows in chanmaster for each row in rosmaster that it matches.
   query table=chanmaster,rosmaster fields=a.standard offset=a:b:5 distinct
shows only the unique rows in chanmaster that participate in the query. Similarly
   query table=chanmaster,rosmaster offset=a:b:5 distinct \
      fields=b:standard
shows the rows in rosmaster. Reversing the order of the tables would have had the same effect.

Distinct can also be used on single table queries. E.g., to get a list of ROSAT PI's

   query table=rosmaster field=pi_lname distinct

Divider

The separator that is used in a CSV/TSV/... style upload file. The default is a comma. The presence of this keyword indicates that uploads are in the CSV/... style. The special string "\t" may be specified to indicate tab dividers. Other dividers may need to be escaped appropriately for the users shell.

Examples

   divider='\t'
   divider='||'

EndTimeOptional

This keyword is used when a temporal query is made of a table that has start and end time fields, but where the end time may sometimes be null. In that case the start and end times will be considered to be the same. Otherwise all such rows will be eliminated from the results. This is commonly needed when querying master tables.

Fields

This keyword specifies the output rows to be shown. Possible values are
Standard
All standard rows for all participating tables.
X:Standard
All standard rows for table with alias X.
All
All rows of all participating tables.
X:All
All rows for table with alias X.
Column names
The name of a column (possibly including an alias)
Expression
An SQL expression
By default the standard fields for the first table are shown.

Examples

   fields=standard
Show the standard columns for all tables.
   fields=a:all,c:standard
Show all columns for the first table, none from the second and the standard columns from the third
   fields=name,ra,dec
Show the name, ra, and dec columns from the first table that has each field -- they migth even come from separate tables.
   fields=a.name,a.ra,a.dec,b.name,b.ra,b.dec,a.ra-b.ra,a.dec-b.dec
Show the given columns and expressions.

Format

The output format for the table. Currently supported formats are
VOTable
The VOTable XML output.
Stream
A pipe delimited format which does not adjust the size of the fields to a constant allowing results to be streamed as they are received from the server
Aligned
a pipe-delimited format similar to the current Batch format. The columns are padded to ensure that all columns are the same width.
Excel
An Excel XLS file.
FITS
A FITS binary table
Meta
An XML format designed for metadata queries.

Example

    Format=votable

Generic

Include generic products. Generic products are products that are not specifically linked to a table but can be linked to any table that has the appropriate data fields (typically RA and Dec). This includes NED, SIMBad and SkyView image services.

HeaderLines

The number of header lines in the CSV/TSV/... files. The first header line is assumed to be the column names separated by the specified (or default comma) divider. The remainder of the header lines are ignored.

Example

    HeaderLines=3

Help

Display help. With no arguments the entire help file is displayed. The values Summary, Overview and Examples as well as the individual setting keys may be used to get more specific help.

Examples

   Help
   Help=summary
   help=format
If the help setting is seen, no further processing is done.

MyTables

List the tables belonging to the user (the user and possibly password settings are required).

Example

   User=someuser mytables

Offset

This indicates that only rows where the difference in position between the two tables specified is less than the given value should be returned. The participating tables are indicated by their aliases. The three tokens in the offset are separated by colons.

Example

    table=rosmaster,ascamaster offset=a:b:.01

This example does a spatial cross-correlation between rosmaster and ascamaster with a tolerance of 0.01 arcminutes. In principle a constraint could be used to specify this, but it is quite tedious to write out. If the third element of the offset triad is omitted, a default radius based on the first table is used. If ShowOffsets also specified, then a column will be generated giving the offset distance.

OuterJoin

An outerjoin may be desired when you wish to query a table and include information from it, but not exclude results from a primary table even if there is no match in the secondary. E.g., suppose we have a table of objects some of which have been observed by Chandra. We may wish to include the Chandra exposure time when there is a Chandra observation. A simple join between the object table and the Chandra master table would exclude all objects not observed by changed. We can use outer joins to get around this restriction. The syntax of this value is
   outerjoin=alias:constraint[:type]
where the alias is the name of the table with which the outer join is being made, constraint includes all of the constraints of the table the join is being made against and type should be one of LEFT, RIGHT or FULL. It defaults to LEFT.

Outer joins are used in Xamin for index queries. A given table may or may not have a regime, mission or contain specific object types. We want to be able to perform queries that gather this optional information but include all tables regardless of whether it is present.

Example:

    tables=myobs,chanmaster
    outerjoin=a:a.obs_no=b.obsid
might be appropriate if our table has a field a.obs_no which contains the Chandra observation number or -1 if never observed.

Password

Specify the password for the given user when the user account requires a password A password is required whenever the user keyword is required.

Position

This is used to specify a search around one or more positions in the sky. Positions may be specified as targets or coordinates. More than one position may be specified and separated by semicolons within a single setting, or you can specify multiple position settings.

If more than one table is participating in the query, then it is assumed that a position refers to the first table with ra and dec fields.

If both a single position and showOffsets are specified, then an offset column will be shown. However if you want to see the offset for multiple inputs, then a source list should be used.

Examples

    position='10 13 47, 9 18 22'
    position='3c273;10 19 43, 10 14 03.1;9.27 14.24'
    position=3c273 position='a1656;a2156'
The first example specifies a single position. The second specifies three different positions to be queried. The last also specifies three different positions.

Products

Include products information for the specified rows if products are supported for the chosen output format. This may be specified without a value in which case all top level data products for the requested tables are shown, or the user may specify a list of products to request. If an element in the list includes a dot (e.g. rosat.obs_pspc), then that specific product is searched for. If not, then the top level products in that hierarchy are looked for. E.g., products=rosat.obs_pspc,asca searches for a particular ROSAT product and any top level asca data product (a cross-correlation may have products from multiple missions).

Examples:

    products
    products=rosat.obs_im,link.ascamaster,generic

Resolver

Specify the name resolver or resolvers to be used in converting target names to coordinates. Resolvers include SIMBAD, NED and GRB. The default is to search all of those in that order and return the first match.

ResultMax

This argument gives the maximum number of rows to be returned. Normally the query will give an indication of whether additional rows are available if the limit is reached. (Internally the program asks for one more row that you do just in case your limit would match the number of rows returned with no limit). Values less than or equal to 0 are treated as no limit.

Example:

   ResultMax=10

SaveAs

Save the query result as a table in the current space. An existing table cannot be overwritten (use Delete to remove it first). The name of the table does not include the user name prefix. After a table has been saved it can be used in subsequent requests as user.name. If the table already exists the results of the current query will be appended to the exising table so long as the output fields are compatible.

Example:

    user=someuser saveas=mytable

ShowAnti

If an anticorrelation is being performed, then show the name of the table being anticorrelated against as the first column in the result. This is most commonly useful when the anticorrelation is being done against a union. This allows the user to effectively do anticorrelations against multiple tables in a single query while retaining information in the results about which table was NOT matched for each row of the results. Example:
    showAnti

ShowCost

This keyword (whose value is not used) specifies that a cost estimate (in CPU time) for the query should be given.

Example:

   showcost

ShowOffsets

This keyword has no value. It adds columns to the end of the table which give the offsets between the tables in cross-correlations and the positional offset in a cone search if there was only a single position specified.

ShowSQL

This keyword (whose value is not used) specifies that the SQL used in the query should be shown.

SkipQuery

This keyword specifies that the query should not be run. Normally it would be used in conjunction with ShowCost or ShowSQL. It can also be useful if debugging a complex query since any expressions used in the query will be parsed and checked even though the query is not actually run.

SortVar

This argument gives one or more expressions that should be used for sorting the results. While sort fields are normally columns, arbitrary SQL expressions may be used.

Note that if you are applying a row limit to the query (see ResultMax), then the entire query must be run and sorted before the first N rows are output.

A '+' or '-' may be suffixed to the expression to indicate an ascending (default) or descending order to the sort.

Instead of an expression, you can also give the column index to be queried. This may be required if you are doing a union query. The first column is indexed as 1.

Examples:
    sortvar=dec-,ra
    sortvar=abs(a.ra-b.ra)-
    sortvar=2
The first example sorts the results from +90 to -90 degrees in declination. When two rows have the same declination the smaller RA is shown first.

The second example might sort a cross-correlation with the rows with the largest deviations in ra shown first.

The last example sorts by the second column in the output.

Table

Table is used to define one or more tables to be used in the query. Note that if multiple tables are specified, then a query is made of the join of the specified tables, not separate queries of each.

Example

   table=rosmaster,ascamaster
When multiple tables are included in a query, the tables and their columns are referred to by aliases. The alias for the first table is 'a', the second is 'b', and so forth. E.g., in the example above if we wish to place a constraint on the ASCA declination we would refer to it as b.dec.

Time

Time is used to specify a general time constraint on the query. Times are specified as interval1,interval2,... where the interval may be given as an instant or a range with two instants separated by the .. operator. An instant may be specified in either ISO format or as a JD or MJD time. The ISO format is somewhat flexible. Years may be specified as either two or four digits (years > 66 are assumed to be prefixed with 19, while years <= 66 are prefixed with 20). The months, days, hours, minutes and seconds may have either one or two digits. Seconds may have a decimal component. Seconds are optional when a time is specified. Note that the accuracy depends uses the Unix standard time conventions and is unlikely to accommodate leap seconds accurately.

Julian dates are assumed for numbers > 1000000 while MJD is used for smaller numbers.

When the user specifies an instant, the time range is expanded in both directions using the value of the timedelta setting. E.g., if you specify Time=24093 with timedelta at its default value of 1, the search will look for all data overlapping the range 24092.0 to 24094.0.

Some tables have a single time value rather than specifying an explicit range. For these tables matches occur when the table specified instant is within the user specified range. When the table specifies a range, a row is included if there is an overlap between the table interval and the user interval.

Examples:

   time=50129.1..50289              An MJD range
   time=2450129.6..2450289.5        The corresponding JD range
   time=2000-10-1..2000-11-23       An ISO range
   time='00-10-01 10:12..00-10-01 10:15'  A three minute range in
             October of 2000.
   time=50129.283                   An MJD instant
   time=2004-12-19                  An ISO instant
   time=50029,50068,50093..50098    Three times.

TimeDelta

Padding added to to both sides of a time instant when the user specifies a single time and not a time range. TimeDelta may be set to 0 which will return only rows where the time is within the range specified in the table row or is equal to the time specified in the table row when only a single time is given. The unit of TimeDelta is days. Note that the padding is added to both sides so that the total searched range is twice as large. Example:
   time=2000-10-10T12:00 timeDelta=0.041667
This will match between 2000-10-10T11:00 and 2000-10-10T13:00

Union

A list of tables to be substituted for the UnionTable in a union query. The result of the query is the concatenation of all rows returned from all of the queries. Only columns that are common to all tables in the union (including the UnionTable) may be used in the query.

Unions may be used with anticorrelations but there are some limits. For a simple anticorrelation (A^B), both the source table and the anticorrelating table may be a union. However if more than two tables are involved in the anticorrelation not counting the union (AxB ^ C), then only the anticorrelating table may be a union.

If the anticorrelating table is a union, then the behavior of the anticorrelation is controlled by the presence of the ShowAnti setting. If this is not specified, then the union is performed before the anticorrelation is done. E.g., rows are returned only if no match is found in any of the tables in the union. If the ShowAnti keyword is shown, then a row is returned for each table in the union for which no row is found. E.g., if the setting is not present then the query is symbolically performed at A^(B|C|D) but if set, the query is (A^B)|(A^C)|(A^D). Example:

    union=rosmaster,chanmaster,xmmmaster

UnionTable

This keyword indicates which table is to be replaced by the tables mentioned in the Union setting. If the UnionTable is not given, then the first table in the query is used. Example:
    uniontable=ascamaster

Upload

A file or list of files to be uploaded as temporary tables which may then be used within the query. Upload files may be VOTables, TDAT files or CSV/TSV/... style. If the files name ends with .tdat, then TDAT format is assumed. If the file name ends with .csv or the divider setting is specified then CSV/...format is assumed. Otherwise VOTable format is assumed.

For CSV files the format of the columns is inferred from the data. Columns whose values can be converted to integer are assumed to be integer are assumed to be of that type. Float is tried next, and if neither of these works the column is assumed to be a text string.

Fields in uploaded tables may be referenced in the query using aliases just as tables specified in the table setting. The aliases for the uploaded tables are always after the preexisting tables. E.g., in a query

     upload=sources.list,mydata.fits table=rosmaster
we have a three way correlation where the fields of the ROSMASTER table will use the alias a, the fields in the source list, sources.list, will used b, and the table info in the first extension of mydata.fits will use the alias c. Examples:
   upload=c:\file1.vot
   upload=../mydata/mylist.csv
   upload=tab1.vot,tab2.vot

User

Specify the user when actions requiring persistence are requested. The settings SaveAs, Delete, MyTables require. The appropriate password setting must also be given.

Example:

   user=someuser