Transportable Database Aggregate Table (TDAT) Format
The TDAT file format is used to serialize HEASARC database tables with the
data and metadata combined into a single, simple, structured file. TDAT files
comprise a sequence of standard ASCII characters with the following
... keyword=value header lines
... data lines, 1 line per record usually using a vertical bar ('|')
character as a field delimiter
Comments may be placed in the <HEADER> section. Comment lines are lines
in which the first character(s) is(are) "#" or "//". Current software systems
may allow arbitary text before the <HEADER> or after the <END>
specifier, but only comments should be placed there.
TDAT Structure Keywords
The above keywords are case-insensitive, but the uppercase versions are
- This required keyword designates the beginning of the header
information. It should be the first non-comment token
in the file and should be placed by itself on a line.
- This required keyword flags the end of the header information and the
beginning of the data section of the TDAT file.
It should be placed by itself on a line.
- This optional keyword specifies that the data section has ended and
further processing of the file is not needed. Any text that comes
after this keyword is ignored. It should be placed by itself
on a line.
All header information is in "name=value" format. The order of parameter
definitions in the header is immaterial. Blank lines and comment lines (lines
that start with "#" or "//") are ignored. Many of the parameter values (the
portion of the parameter definition that comes after the equal sign) can be
encased in quotes. Double quotes, single quotes, and back quotes are all
recognized, but each pair of quotes must match in type. Whitespace on either
side of the equal sign is ignored, as is any whitespace at the beginning of a
line or trailing at the end of a line. Non-blank lines must be either comment
lines or parameter definitions in the "name=value" format with exactly one
such definition per line. Continuation lines are not supported.
Standard Header Keywords
Here is the list of recognized "name=value" parameters:
- table_name = origin_tablename
- Required. The supplied table name must either be a recognized system
table ("zzgen", "zzext", "zzpar", and "zzrel") or begin with "origin_" where
"origin" is the originator of the database. Currently, the only recognized
origin is "heasarc"; however, that is easily changed. If an unrecognized
origin is specified, the program will issue a warning but will continue with
the ingest procedure anyway. Maximum length of a table name (including the
"origin_" prefix) is 20 characters. Anything greater than that will be
truncated and a warning issued. This value is stored in the
- table_description = 'Here is the description.'
- Optional, but strongly recommended. This is the description that goes
into the metabase. This is stored in the zzgen.table_description field.
Maximum description length is 80 characters. Anything greater than that will
be truncated and a warning issued.
- table_document_url = http://heasarc.gsfc.nasa.gov/W3Browse/observatoryname/tablename.html
- Optional, but strongly recommended. This value is intended to
be a URL for a document that describes the database table and its fields.
This field is stored in the zzgen.table_document_url field.
- table_security = private|public
- Optional. The default is public. If table_security is public, then
the software automatically grants select permissions to the public.
- field[field_name] = type[:fmt][_unit] [[ucd]] [(index)|(key)] // description [// comment]
- One entry required for each field in the table. Each line
will be used to populate a row in the ZZPAR table. There are many
(often optional) subfields in this line which are as follows:
The combination of type and format is stored in the zzpar.parameter_format
field. This has a maximum of 24 characters and it is an error to exceed
- This field is specified on the keyword side of the '='. It
indicates the parameter name whose characteristics are being given
in the line. the field name must be shorter than 24 characters.
- The type gives the basic storage format for this field.
Recognized types are listed below. Unrecognized types are
flagged as an error. The following are the supported field types.
Recommended types are given in bold.
int1 -- a one-byte integer
integer1 -- " " "
tinyint -- " " "
int2 -- a two-byte integer
integer2 -- " " "
smallint -- " " "
int4 -- a four-byte integer
integer4 -- " " "
integer -- " " "
float4 -- a four-byte floating point number
real -- a synonym for 'float4'
float -- an eight-byte floating point number
float8 -- " " " " "
char(#) -- character string, # must be in the range 1 to 2000
char# -- character string, # must be in the range 1 to 2000
- The fmt field is optional. It is signaled by a ':' after
the type (i.e., without any separating whitespace).
The fmt specifier is only allowed for floating-point and
integer types. It allows one to specify the way floating-point
values should be displayed. It uses the C or Perl printf()
syntax, but without the leading percent symbol.
- The unit specifier is also optional and is signaled by a '_' after
the type (or type:fmt). It is used to indicate the units
that are used in the field. The unit is stored in the
- The Unified
Content Descriptor (UCD) of the parameter. It is
optional, but strongly recommended. The UCD is specified with
square brackets around it, and it is case-sensitive. The UCD is
stored in the zzpar.parameter_ucd field.
- (index) or (key)
- The index and key flags are optional but mutually exclusive.
The "(index)" flag tells the program to create an index on this
field after the data has been ingested into the database, while
the "(key)" flag suggests that this field be used as the primary key
for storing the database. If multiple key fields are specified
then only the first one (alphabetically) may be used if the
database system does not support multiple fields in the primary key
index. If key is specified, the zzpar.parameter_is_index value is
set to 'K'. If index is specified, it is set to 'Y'. Otherwise, it
is set to 'N'.
- The description is optional but strongly recommended. It gives the
information that will be stored in the zzpar.parameter_description
field and used to tell database users what this parameter means.
It is truncated to a maximum of 80 characters.
- The comment field is optional and is used to attach some additional
comment to the database. It is stored in the zzpar.parameter_comment
field. Generally, it is used to provide additional information
about field values. (E.g., "If type XP: pulse period of the
pulsar"). It is truncated to a maximum of 80 characters.
Here are some example field definitions:
field[ra] = float8:.5f_degree (key) // Right Ascension
field[dec] = float8:.5f_degree (key) // Declination
field[flux]= float4:.2e_mJy (index) // Flux Density Measured by Detector
field[time]= int4_mjd (index) // Start Time of Observation
- relate[field_name] = other_table_name(other_table_field_name) // description
- Optional and obsolete. This definition specifies that a relationship
exists between the supplied field_name and
other_table_field_name of other_table_name. The optional
description is strongly recommended. If either the table
other_table_name or the parameter other_table_field_name does
not exist, then a fatal error is issued, so make sure they exist first.
- parameter_defaults = field_name1 field_name2 ...
- Optional. This parameter definition designates the default column
order for output in, for example, Browse. Not all field names have to be
listed. This information is stored in the zzpar.parameter_defaults column.
The first parameter listed is given the value 1, then next 2, and so forth.
Parameters not specified in this list are given the value 0.
- line[line_number] = field_name1 field_name2 ...
- Required. This specifies the order of the fields on a given
line_number in the data portion of the file. If a field name is specified
here that was not also defined using a "field[field_name]=..." definition,
then a fatal error is issued. The syntax allows the data to be given
using multiple lines in the <DATA> section; however, it is recommended
that only a single data line be used for each record input, i.e., that only
line be specified.
- record_delimiter = "record_delimiter_string"
- Optional and deprecated. Default is no record delimiter at all.
The following list standard escaped character sequences and their equivalent
meanings can be used: \t (tab), \b (backspace), \r (carriage return), \f
(form feed), \v (vertical tab), \a (audible alert/bell), and \### (where ###
is a number between 1 and 127 and represents the ASCII character with that
numerical code). Note: Specifying a record delimiter value of "" is
interpreted as a single blank line between records.
- field_delimiter = "field_delimiter_character(s)"
- Optional and deprecated. Default is the "|" character. The last field
on the line should be delimited. Feature: You can specify more than just one
character as field delimiters. For example, if the data file uses both "|"
(vertical bar) and "!" (exclamation point) as delimiters in different places,
you can set 'field_delimiter = "|!"' and both characters will be interpreted
as field delimiters. The field_delimiter values can use the same escaped
character sequences as used in record_delmiter values. For example,
'field_delimiter = "|\t"' would interpret both the vertical bar character and
the tab character as valid field delimiters for the data. Note: Although you
can specify a field delimiter other than the "|" character, "|" characters
may not be used in any fashion other than as field delimiters.
Any other "keyword=value" pairs other than those given above are treated
as virtual parameters.
The case of most parameter definitions does not matter. Most are converted to
lowercase internally. The only exceptions are the following: field
descriptions, field comments, the table_description value, the
table_document_url value, virtual parameter values (as described in the
previous paragraph), the record_delimiter value, and the field_delimiter
- virtual_parameter_name = virtual_parameter_value
- For example, 'observatory_name = ASCA'. The HEASARC has come up with
a few local conventions for virtual
parameters. For example, virtual parameter values that refer to database
fields, the database field name is preceded by the "@" symbol. Case does not
matter for virtual parameter names. All virtual parameter names are
automatically converted to lowercase by the HDBingest software.
The data format is simply a sequence of lines which have record delimiters
and field specified as above. Note that for character fields,
leading spaces are significant. but are ignored for numeric fields. Since
floating point fields are stored as ASCII values, it should be recognized
that there may be small changes due to rounding error when these data are
transferred among databases. Specifying the display format for floating point
fields is therefore strongly recommended.
Null data fields for parameters are indicated by a zero-length field.
An example TDAT file for the Messier catalog is given below. This example
was generated by the HDBexgest program, but unless otherwise indicated
other software may wish to provide header data in the same order.
Only a few of the data records are included for the sake of brevity.
# TABLE: heasarc_messier
# TOTAL ROWS: 109
table_name = xx_messier
table_description = "Messier Nebulae Catalog"
table_document_url = http://heasarc.gsfc.nasa.gov/W3Browse/general-catalog/messier.html
table_security = public
# Table Parameters
field[alt_name] = char10 (index) // Alternate designation
field[bii] = float8_degree (index) // Galactic Latitude
field[class] = int2 (index) // Browse Object Classification
field[constell] = char4 (index) // Constellation of Origin
field[dec] = float8:.4f_degree (index) // Declination
field[dimension] = char6_arcmin (index) // Dimensions of the Source
field[lii] = float8_degree (index) // Galactic Longitude
field[name] = char6 (index) // Source designation
field[notes] = char50 (index) // Notes
field[object_type] = char2 (index) // Object Category
field[ra] = float8:.4f_degree (index) // Right Ascension
field[vmag] = float4:4.1f (index) // Visual Magnitude
field[vmag_uncert] = char2 (index) // Magnitude Uncertainty
parameter_defaults = name alt_name ra dec constell dimension vmag vmag_uncert class
# Virtual Parameters
declination = @dec
default_search_radius = 60
equinox = 2000
frequency_regime = Optical
observatory_name = GENERAL CATALOG
right_ascension = @ra
table_priority = 3
table_type = Object
target_name = @name
unique_key = name
# Relationship Definitions
relate[class] = heasarc_class(class_id)
# Data Format Specification
line = alt_name bii class constell dec dimension lii name notes object_type ra vmag vmag_uncert
NGC 6809|-23.2733634|3080|SGR|-30.9666947708543|19|8.7909942|M 55||GB|294.99980605110801|7.0||
NGC 6715|-14.0974428|3080|SGR|-30.483349176838999|9|5.6077231|M 54||GB|283.77480378105798|7.7||
NGC 6121|15.9648287|3080|SCO|-26.5333061473825|26|350.9692185|M 4||GB|245.89981204924899|5.9||
NGC 1904|-29.2898547|3080|LEP|-24.550010243430599|9|227.2840613|M 79||GB|81.124872363673802|8.0||
NGC 2447|.1495137|3600|PUP|-23.866637331244299|22|240.0587793|M 93||OC|116.149868339422|6.2|:|
NGC 7099|-46.8435102|3080|CAP|-23.1833879909756|11|27.1771312|M 30||GB|325.09982631146403|7.5||
NGC 6531|-0.43694830|3600|SGR|-22.500001387839401|13|7.7120967|M 21||OC|271.149814658205|5.9||
NGC 2287|-10.226592|3600|CMA|-20.733319763806499|38|231.0978149|M 41||OC|101.74986693951899|4.5||
NGC 6494|2.872862700|3600|SGR|-19.016665704498902|27|9.8346853|M 23||OC|269.199819080682|5.5||
Documentation prepared by the
HEASARC Database Group
HEASARC Home |
Last modified: Monday, 09-Oct-2006 20:11:06 EDT