OGIP Calibration Memo CAL/GEN/2004-001

Specifying the Location of Defects in Charge Coupled Devices: The BADPIX File Format

M. F. Corcoran
Mail Code 662,
NASA/GSFC,
Greenbelt,
MD 20771.

Version: 2004 November 15

1 SUMMARY

This document describes ways to specify the location and type of defective pixels in CCD detectors. It is based on file formats currently in use by the CHANDRA and XMM-Newton missions.

Intended audience: primarily OGIP programmers, hardware teams & authors of analysis s/w.

2 LOG OF SIGNIFICANT CHANGES

Release	Sections Changed	Brief Notes
Date

2004 Nov 09		Original Version
2004 Nov 15	§ 6 & 7	added
2004 Nov 15	§ 5.4	WMAP HDUVERS 2 compatible

1 SUMMARY
2 LOG OF SIGNIFICANT CHANGES
3 INTRODUCTION
4 The Bad Pixel List Format
    4.1 EXTNAME
    4.2 HDUCLASn keywords
    4.3 OGIP CALDB Keywords
    4.4 The BADPIX Table Columns
        4.4.1 Required Table Columns
        4.4.2 Optional columns
    4.5 Sample Header
5 BAD PIXEL Image Masks
    5.1 EXTNAME
    5.2 HDUCLASn keywords
    5.3 OGIP CALDB Keywords
    5.4 Coordinate Specifications
    5.5 Image Values
6 Connecting Bad Pixel Lists and other files
7 References

3 INTRODUCTION

Charge Coupled Devices, or CCDs, are semiconductor devices that convert patterns of light into patterns of electric charge. CCDs are composed of arrays of picture elements, or pixels which convert incoming photons into electric charges which are then read out to provide information on the time of arrival, location, and energy of the incoming photon. CCDs have numerous advantages over other X-ray astronomy detector technologies. For example CCDs are solid-state devices so they are not prone to gas leaks which can limit the lives of proportional counters. CCDs provide energy sensitivity along with positioning, unlike micro-channel plates. CCDs have other advantages: linear sensitivities, high quantum efficiencies, and high signal to noise ratios.

CCDs can be front-illuminated, in which incoming photons are detected on the top surface of the CCD, or back-illuminated. Back illuminated CCDs generally have higher quantum efficiencies but are more difficult to build since their substrate surfaces have to be very thin. The use of CCDs as detectors in astronomical X-ray observatories began in the early 1990s with the ASCA satellite. ASCA had 2 CCD detectors known as the Solid-State Imaging Spectrometers (SIS) constructed collaboratively by the Massachusetts Institute of Technology, the Institute of Space and Astronautical Science (ISAS) of Japan, and Osaka University. Each SIS detector consisted of four 420×422 square pixel array CCD chips. Each SIS operated in the 0.4 keV to 10 keV range, with an energy resolution of about 2 percent at 5.9 keV and a field of view of 22¢×22¢. Both CHANDRA and XMM-Newton use CCDs as their primary detectors.

CCD detectors have some disadvantages. Thermal electrons produce "dark-current" noise. Bright sources can saturate neighboring pixels, producing bright streaks in the output image. Multiple events which arrive at a single pixel within a single readout cycle are counted as a single photon, a phenomenon known as "photon pile-up". Also, individual pixels in a CCD array may lose sensitivity or may be affected by excessive noise. It is usually necessary to know the location and properties of such pixels to avoid confusion with real sources.

This document describes a file format for specifying the locations and properties of defective pixels in a CCD detector. It is based on formats currently in use, primarily by the CHANDRA and XMM-Newton data centers.

4 The Bad Pixel List Format

The Bad Pixel list is a tabulation of the locations of CCD detector defects. These locations are in general specified as rectangular regions which give the start position and the ending position (in appropriate DETECTOR coordinates) of the defective pixels. This list should be provided as a binary table (BINTABLE) in a standard FITS file and should include the following information.

4.1 EXTNAME

The extension name keyword EXTNAME should have the value BADPIX, i.e.

EXTNAME = 'BADPIX  '           / name of this binary table extension

4.2 HDUCLASn keywords

The HDUCLASn keywords provide a mechanism for software to identify relevant header-data units (HDUs) in FITS files. For Bad Pixel Lists, these keywords should be included in the appropriate HDU and have the following values, as given in table 1

Table 1: HDUCLASn Keywords for BADPIX Tables

Keyword	Defined Values	Description
HDUNAME	BADPIX	name of this HDU
HDUDOC	CAL_GEN_2004_001	Document describing the file format
HDUCLASS	OGIP	indicates the organization which devised the file format in use
HDUCLAS1	REGION	indicates data specifies a region
HDUCLAS2	DETECTOR	indicates the region data is in detector coordinates

4.3 OGIP CALDB Keywords

If the bad pixel list is to be used as a calibration database (CALDB) file, the mandatory CALDB keywords must be included in the extension header, along with optional CALDB keywords if necessary. These keywords are listed in table 2:

Table 2: CALDB Keywords and Values

Header Keyword	Description	Value
TELESCOP	The name of the satellite or mission	Standard Values are given in OGIP/93-013
INSTRUME	Name of the instrument	Allowed values given in OGIP/93-013
DETNAM	Name of the detector (optional). Use if INSTRUME value is ambiguous	Standard values given in OGIP/93-013
FILTER	(optional) Name of the filter in use, if applicable	Mission-specific
CCLSmmmm	HEASARC class of the calibration file	BCF
CDTPmmmm	Distinguishes "real" or "virtual" calibration information	DATA
CCNMmmmm	Contents Descriptor	BADPIX
CBDnmmmm	Parameter limits for a given calibration dataset	Mission Defined. See Section B.2 of CAL/GEN/92-011
CVSDmmmm	Date (UTC) after which calibration dataset can be used	Mission-defined
CVSTmmmm	Time (UTC) of day on specified CVSDmmmm after which calibration dataset can be used.	Mission-defined
CDESmmmm	Descriptive summary of calibration dataset	Mission-defined

4.4 The BADPIX Table Columns

The FITS binary table which contains the BADPIX information should have the following columns. The order of the columns is not important.

4.4.1 Required Table Columns

The BADPIX table must at minimum specify the location of the chip defect in appropriate detector coordinates

TTYPE: CHIPX, RAWX, DETX, etc. (Recommended: RAWX)
FORMAT: Integer (rI), Long Integer (rJ), real (rE), where r is 1 or 2
DESCRIPTION: These values specify the X (Column) position of the defective pixel or region. Positions can be specified as a single number giving the column location of the center of an individual defective pixel. They can also be specified as a 2-element vector specifying the start and stop columns of a continguous region of defective pixels. This coordinate should be given relative to the center of the pixel in the lower left corner, consistent with FITS image conventions. Columns run horizontally and column numbers increase to the right. In general integer pixel values should be sufficient to specify positions of bad pixels, though in some cases BADPIX lists may specify chip gaps whose locations and extents may not be commensurable with the size of a pixel.

TTYPE: CHIPY, RAWY, DETY, etc. (Recommended: RAWY)
FORMAT: Integer (rI), Long Integer (rJ), real (rE), where r is 1 or 2
DESCRIPTION: These values specify the Y (row) position of the defective pixel or region. Positions can be specified as a single number giving the row location of the center of an individual defective pixel. They can also be specified as a 2-element vector specifying the start and stop rows of a continguous region of defective pixels. This coordinate should be given relative to the center of the pixel in the lower left corner, consistent with FITS image conventions. Rows run vertically and row numbers increase to the top. In general integer pixel values should be sufficient to specify positions of bad pixels, though in some cases BADPIX lists may specify chip gaps whose locations and extents may not be commensurable with the size of a pixel.

4.4.2 Optional columns

The BADPIX list can also contain a definition of the region shape associated with each row in the list. Usually this is a rectangular region on the CCD chip marked by the pixel positions of the lower left and upper right corners of the rectangle. Alternative means of specifying include simply listing pixels individually, or listing the pixel location of a bad row or column, along with the length of the row or column.

TTYPE: SHAPE
FORMAT (TFORM): CHARACTER (nA)
DEFINED VALUES: "RECTANGLE" or "POINT"
DESCRIPTION: This column specifies the shape of the defective pixel region. If the value is "RECTANGLE', the regions are specified by four numbers giving the starting and ending columns (the X coordinate) followed by the starting and ending lines or rows (the Y coordinate). This convention follows the Image Reduction and Analysis Facility (IRAF) convention for specifying bad pixels. If the value is "POINT", then the defect can be specified by two numbers, the first giving the column position (the X coordinate) followed by the line (or row) position (the Y coordinate). Coordinates are specified in DETECTOR coordinates, i.e.the region boundaries are specified in a coordinate system tied to the CCD chip. Usually this coordinate will have at its origin the center of the pixel in the lower left corner, consistent with FITS image conventions. Columns run horizontally and column numbers increase to the right. Rows run vertically and row numbers increase towards the top of the CCD chip. This convention is used by the CHANDRA mission.

TTYPE: YLENGTH
FORMAT: Integer (1I) or Long Integer (1J)
DESCRIPTION: Length of bad column ( YLENGTH=1 indicates a single bad pixel). This convention is used by the XMM-Newton mission.

TTYPE: XLENGTH
FORMAT: Integer (1I) or Long Integer (1J)
DESCRIPTION: Length of bad row ( XLENGTH=1 indicates a single bad pixel).

TTYPE: STATUS (Alternative: DESCRIPTION)
FORMAT: Mission-specified
DESCRIPTION: This column can be used to specifies the type of defect. Examples of types of defects include pixels which are insensitive ("dead"), randomly noisy ("flickering"), always on ("hot"), or a space between pixel rows or columns ("gap"). The STATUS code used should be defined either in the header of the BADPIX HDU or in a separate extension in the same file.

TTYPE: TIME
FORMAT: Mission-specified
DESCRIPTION: This column can be used to specify the time at which a pixel became defective, i.e. a pixel is considered defective for all times ³ TIME. Time can be specified in any convenient unit: UTC, mission elapsed seconds, etc. The TIME system used should be documented in the header of the BADPIX HDU.

TTYPE: CCDID (Alternative: CCD_ID)
FORMAT: Mission-specified
DESCRIPTION: This column can be used to identify the CCD chip if a detector consists of a multi-chip array for a particular entry in the BADPIX list. If the BADPIX list contains pixels from one CCD only (i.e. all entries in the CCDID column are identical) this column may appear as a HEADER keyword in accord with the Greenbank convention.

TTYPE: NODE_ID (Alternative: NODE)
FORMAT: Mission-specified
DESCRIPTION: This column gives the CCD node for a particular entry in the BADPIX list. If the BADPIX list contains pixels from one CCD node only (i.e. all entries in the NODE_ID column are identical) this column may appear as a HEADER keyword in accord with the Greenbank convention.

TTYPE: MODE_N/A
FORMAT: Mission-specified
DESCRIPTION: This column gives the particular CCD modes for which the BADPIX information is not applicable. It is recommended that this information be provided by the CALDB boundary keywords, but is included here for consistency with XMM-Newton bad pixel tables.

TTYPE: N_Sat_IMP
FORMAT: Mission-specified
DESCRIPTION: This column gives the brightness of a particular hot pixel, and is included here for consistency with XMM-Newton bad pixel tables.

4.5 Sample Header

The following is adopted from an ACIS bad pixel file.

XTENSION= 'BINTABLE'           / binary table extension
BITPIX  =                    8 / 8-bit bytes
NAXIS   =                    2 / 2-dimensional binary table
NAXIS1  =                   36 / width of table in bytes
NAXIS2  =                   88 / number of rows in table
PCOUNT  =                    0 / size of special data area
GCOUNT  =                    1 / one data group (required keyword)
TFIELDS =                    6 / number of fields in each row
EXTNAME = 'BADPIX  '           / name of this binary table extension
EXTVER  =                    5
HDUNAME = 'BADPIX5 '           / ASCDM block name
LONGSTRN= 'OGIP 1.0'           / The HEASARC Long String Convention may be used.
COMMENT   This FITS file may contain long string keyword values that are
COMMENT   continued over multiple keywords.  The HEASARC convention uses the &
COMMENT   character at the end of each substring which is then continued
COMMENT   on the next keyword which has the name CONTINUE.
CCD_ID  =                    5
TTYPE1  = 'SHAPE   '           / region shape
TFORM1  = '16A     '           / format of field
TTYPE2  = 'COMPONENT'          / Component number
TFORM2  = '1I      '           / format of field
TLMIN2  =                    0
TLMAX2  =                   36
TTYPE3  = 'CHIPX   '           / CHIP X location
TFORM3  = '2I      '           / format of field
TUNIT3  = 'pixel   '
TLMIN3  =                    1
TLMAX3  =                 1024
TTYPE4  = 'CHIPY   '           / CHIP Y location
TFORM4  = '2I      '           / format of field
TUNIT4  = 'pixel   '
TLMIN4  =                    1
TLMAX4  =                 1024
TTYPE5  = 'TIME    '           / Time pixel went bad
TFORM5  = '1D      '           / format of field
TUNIT5  = 's       '
TTYPE6  = 'STATUS  '           / Badpixel status code
TFORM6  = '16X     '           / format of field
TUNIT6  = 'coded   '
CONTENT = 'BADPIX  '
HDUCLASS= 'OGIP    '
HDUCLAS1= 'REGION  '
HDUCLAS2= 'DETECTOR'           
ORIGIN  = 'ASC     '           / Source of FITS file
CREATOR = 'acis_build_badpix - Version CIAO 2.3' / tool that created this output
REVISION=                    1
ASCDSVER= '6.12.0  '           / ASCDS version number
CHECKSUM= '228W227T227T227T'   / HDU checksum updated 2003-05-05T15:46:36
DATASUM = '3388070702'         / data unit checksum updated 2003-05-05T09:15:44
MISSION = 'AXAF    '           / Mission
TELESCOP= 'CHANDRA '           / Telescope
INSTRUME= 'ACIS    '           / Instrument
DETNAM  = 'ACIS-5678'          / Detector
CCLS0001= 'BCF     '           / CALDB class
CDTP0001= 'DATA    '           / CALDB data type
CCNM0001= 'BADPIX  '           / CALDB class name
CBD10000= 'TEMPERATURE(120)C   / CALDB boundary: use when temperature=120C
CVSD0001= '2004-09-01'	      / CALDB: use after this date
CVST0001= '12:15:00'           / CALDB: user after this time on CVSD0001
CDES0001= 'CALDB BAD PIXEL LIST ' / CALDB: description
CALDBVER= '2.21    '           / file included in this version of the CALDB
SIM_X   = -6.8282252473119E-01 / SIM focus pos (mm)
SIM_Y   =  0.0000000000000E+00 / SIM orthogonal axis pos (mm)
SIM_Z   = -1.8332124295204E+02 / SIM translation stage pos (mm)
DEFOCUS =  1.4449365687057E-03 / SIM defocus (mm)
FOC_LEN =  1.0070000000000E+04 / HRMA focal length (mm)
OBS_MODE= 'POINTING'           / Observation mode
DATAMODE= 'FAINT   '           / Data mode
READMODE= 'TIMED   '           / Read mode
FEP_ID  =                    2
END

5 BAD PIXEL Image Masks

It can useful to provide an image of the detector with the bad pixel columns masked out. Therefore bad pixel positions can be provided as a 2-dimensional image rather than as a tabulated list if this is convenient. The image should be given in full (unblocked) pixels. Given the large size of modern CCD detectors, and the small number of expected defective pixels, providing a full mask image is in general not an efficient means of specifying defects. Specification of contiguous regions of defective pixels as a series of small subimages can reduce disk space requirements and can help speed access.

This image should be provided as a primary image or as an IMAGE extension, in a standard FITS file and should include the following information.

5.1 EXTNAME

If the image is an extension, then the extension name keyword EXTNAME should have the value BADPIX, i.e.

EXTNAME = 'BADPIX  '           / name of this binary table extension

5.2 HDUCLASn keywords

The HDUCLASn/HDUVERS keywords provide a mechanism for software to identify relevant header-data units (HDUs) in FITS files. For Bad Pixel Lists, these keywords should be included in the appropriate HDU and have the following values, as given in table 3

Table 3: HDUCLASn Keywords for BADPIX image masks

Keyword	Defined Values	Description
HDUNAME	BADPIX	name of this HDU
HDUDOC	CAL_GEN_2004_001	Document describing the file format
HDUCLASS	OGIP	indicates the organization which devised the file format in use
HDUCLAS1	IMAGE	indicates data is an image
HDUCLAS2	DETMAP	indicates IMAGE is a DETECTOR MAP, i.e. is in detector coordinates

5.3 OGIP CALDB Keywords

If the bad pixel mask is to be included as a calibration database (CALDB) file, the appropriate CALDB keywords must be included in the extension header, along with optional CALDB keywords if necessary. These keywords are listed in table 2.

5.4 Coordinate Specifications

Image coordinates should be specified using the standard World Coordinate System (WCS) keywords. The following keywords defined for "Weighted Maps" (WMAPs), HDUVERS='2.0.0' shown below in Table 4 are recommended. The sample values given would be appropriate, for example, for a 1025×1025 pixel array where the optical axis is at the center of the array, and positions are measured from the pixel in the lower left corner of the array.

Table 4: Recommended Keywords to Specify Coordinate Systems for BADPIX Masks

Keyword	Description	Sample Value
OPTIC1	the X coordinate position of the optical axis in detector coordinates.	513
OPTIC2	the Y coordinate position of the optical axis in detector coordinates.	513
WMREBIN	the rebinning factor from detector pixels to WMAP bins (this should be the single axis binning factor eg a binning factor of 4 puts 16 detector pixels into each WMAP bin).	1 (recommended)
CDELT1	the X axis increment for each WMAP bin in detector units.	1 (recommended)
DDELT1	the original X axis increment	1 (recommended)
CDELT2	the Y axis increment for each WMAP bin in detector units.	1 (recommended)
DDELT2	the original Y axis increment	1 (recommended)
CRVAL1	the coordinate position of the X axis reference pixel.	1
CRVAL2	the coordinate position of the Y axis reference pixel.	1
CRPIX1	the X axis reference pixel.	1
DRPIX1	the original X axis reference pixel	1
CRPIX2	the Y axis reference pixel.	1
DRPIX2	the original Y axis reference pixel	1
CTYPE1	the name of the X axis in the WMAP.	RAWX
CTYPE2	the name of the Y axis in the WMAP.	RAWY
CUNIT1	the unit for the X axis in the WMAP.	pixel
CUNIT2	the unit for the Y axis in the WMAP.	pixel
WCSTY1P	PHYSICAL	PHYSICAL
WCSTY2P	PHYSICAL	PHYSICAL
CTYE1P	Source of X axis.	1
CTYE2P	Source of Y axis.	1
CRPIX1P	X axis reference pixel.	1
CRPIX2P	Y axis reference pixel.	1
CRVAL1P	Coordinate of X axis reference pixel in original image.	1
CRVAL2P	Coordinate of Y axis reference pixel in original image.	1
CDELT1P	X axis increment	1
CDELT2P	Y axis increment	1

5.5 Image Values

In general image values will be either 1 (pixel is good) or 0 (pixel is bad), so that multiplication of the BADPIX mask image and the unbinned data image will result in an output in which bad pixels are set to 0.

6 Connecting Bad Pixel Lists and other files

Analysis of photon event files and image files often require knowledge of an appropriate bad pixel file. We define a keyword named BPIXFILE which can be included in the headers of photon event and image FITS files to connect these files with the appropriate bad pixel file. For example

BPIXFILE='acisf03745_002N001_bpix1.fits'   / bad pixel file for this event list

7 References

The Chandra X-ray Center (CXC) provides a website which lists bad pixels for the ACIS instrument. A list for the HRC is also provided on the CXC calibration website.

A description of XMM-Newton software to generate bad pixel lists can be found in the Calibration Access and Data Handbook, XMM-PS-GM-20 Issue 3.1, by B. Altieri et al., 14 September 2004. In particular the discussion of the output file is helpful in illustrating information which should be provided by bad pixel lists, and it includes a discussion of the information that needs to be included for the XMM-Newton CCDs. The XMM-Newton Science Analysis Software (SAS) documentation page provides a description of the SAS badpix routine, which is used to create XMM-Newton-style badpixel extensions, and the badpixfind routine used to identify bad pixels in XMM-Newton data. The XMM-Newton Current Calibration File Release Notes web page includes documents which describe changes to the badpixel files in the CCF.

File translated from T_EX by T_TH, version 3.85.
On 1 Apr 2011, 17:18.