Skip to main content

Proposals

Effective 8 December, 2005, the IAU (International Astronomical Union) FITS (Flexible Image Transport System) Working Group has approved three proposals to modify the FITS Standard to support the 64-bit integer data type:

  1. Primary arrays or IMAGE extensions containing 64-bit integer data type arrays are designated by BITPIX = 64 (approved conditionally; the exact wording of this change to the Standard is tentative).
  2. Columns in FITS binary tables that contain 64-bit integers are designated by TFORMn = 'nK'
  3. Array descriptor columns in FITS binary tables with 64-bit integer array length and heap offset values are designated by TFORM = '1Qt' (analogous to the existing 32-bit descriptor type designated by TFORM = '1Pt').

The specific changes to the FITS Standard to support these proposals are shown in this marked-up PDF file.

Recent Discussions and Votes

The issues related to adding support for 64-bit integers to FITS were extensively debated on the FITSBITS mail list in April 2005, May 2005, and June 2005.

Much of these discussions centered on whether there is sufficient reason for adding support for 64-bit integer primary arrays and Image extensions to FITS to justify the impact it will have, especially to existing software systems. Here are some of the cited justifications.

The four Regional FITS committees were asked to vote on these proposals in September 2005. The detailed voting results were posted on the FITSBITS newsgroup and are summarized in the following table:

ProposalYesNoAbstain
Images with BITPIX = 643531
Table columns with TFORM = 'K'3900
Descriptor columns with TFORM = '1Qt'3900

The IAU FITS Working Group (IAUFWG) then voted on these same proposals in December 2005. The first proposal (BITPIX = 64) was approved with a 19 "Yes," two "No," and one "Abstain" vote. Under the IAUFWG voting rules, the "No" votes force a 6-month delay before the vote becomes final, in the hope that a compromise can be reached that everyone can support. Hence, the final wording of this change in the FITS Standard is tentative.

The other two proposals were unanimously approved by the IAUFWG and hence became effective on 8 December 2005.

A Brief History of this Proposal

The discussion of adding 64-bit integer support to FITS dates back to at least July 1998, when it was suggested that this should be added to the revision of the FITS Standard that was being drafted by the NOST (NASA/Science Office of Standards and Technology) panel at that time. The NOST panel ultimately decided that it did not have the authority to unilaterally adopt such a major change.

This issue was raised again in 2001 on the fitsbits email exploder (see threads of the discussion in September 2001 and the discussion in October 2001). In an informal straw poll taken in October 2001, there were 20 votes in favor of adding 64-bit integer support to FITS, four votes against, and five undecided or no opinion. It was hard to judge the true sentiment of the FITS community from such a small poll, and the FITS committees that control the FITS Standard took no action at that time. None the less, some software packages began to offer unofficial support for 64-bit integers (see below).

This issue was discussed once again at the FITS Birds-of-a-Feather (BoF) session at the ADASS (Astronomical Data Analysis Software and Systems) meeting held in Pasadena, CA in October 2004. For the first time, there appeared to be wide spread support for this measure after it was confirmed that most existing software environments now support 64-bit integers (even most current generation computers that operate with 32-bit CPU architectures). A show of hands by the ~70 attendees of this BoF session on the question of whether "we should now seriously consider adding support for 64-bit integers to the FITS Standard" showed about 80% in favor, 20% not voting, and only one person against. Given this relatively strong show of support, there was general agreement that this matter should be brought up for discussion by the whole FITS community on the FITSBITS newsgroup and then voted on by the FITS committees.

Sample FITS Files

test64bit1.fit - a small test file that contains:

  1. Primary array containing a small 64-bit integer image with BITPIX = 64
  2. Binary table extension containing a 64-bit integer column with TFORM2 = '1K'
  3. A second binary table extension containing a '1QK' column of variable-length arrays of 64-bit integers.

Existing Software that supports 64-bit Integer FITS Files

  • The JavaFits library developed and supported by Tom McGlynn has supported 64-bit integer images and table columns since its inception in the year 2000.
  • The CFITSIO library has supported 64-bit integer images and table columns since October 2001. Support for 'Q' 64-bit array descriptor columns was added in April 2005.
  • The MWRFITS and RDFITS routines in IDL (Interactive Data Language) have supported 64-bit integer images and table columns since 2002.

Justifications for 64-bit Integer Arrays in FITS Files

Scientific Justifications

The following cases illustrate scientific problems that could require arrays of integer data with more than 32-bits of precision.

  1. Arrays of time measurements where the time is measured as the number of small clock "ticks" from some reference time (e.g., the start times of a series of binary star eclipses).
  2. Histograms generated from data bases of astronomical sources containing more than 2^32 objects.
  3. Rebinned or summed images from large mosaic CCD (charge-coupled device) cameras that may result in more than 2^32 total counts per binned pixel. Similarly, cumulative histograms of 32-bit integer arrays may exceed 2^32.
  4. Arrays of indices to individual pixels in large images (e.g., a list of bad pixels). The indices could involve non-rectangular pixelizations like HTM (Hierarchical Triangular Mesh) or HEALPix so that the total number of pixels matters (they have a 1D index, instead of having separate X and Y coordinate indices).
  5. The precise measurement of time or relative position (using the number of wavelengths of light as the yard stick) in emerging fields such as gravitational wave astronomy and/or interferometry.
  6. Numerical simulations that need more than 32-bits of dynamic range.
  7. Arrays of encryption keys or other numeric codes containing more than 32-bits.

Practical or Sociological Justifications

The following arguments give various practical reasons for supporting 64-bit integer arrays in FITS files.

  1. The 64-bit integer data type is available in virtually every modern programming language. Programmers will inevitably make use of this datatype (whether or not the added precision is strictly required) and will sometimes want to store the results to a FITS file. Having to convert the values to 32-bit integers or 64-bit floats before writing them to the FITS file adds complexity to the software and can lead to errors and confusion if some of the 64-bit integer values are accidentally truncated.
  2. A large segment of the astronomical research community uses FITS as the on-line analysis format for all types of data files, including temporary and/or intermediate numerical results. When data volume constraints are not an issue, programmers may want to save integer results using the data type with the most range and precision, especially if there is any uncertainty about what precision is actually required.
  3. In order to "future proof" applications it may be prudent to use 64-bit integers even if 32-bit integers seem adequate for current needs. Software applications can end up being used for decades, at which point it could be difficult to retrofit the software to support 64-bit integers.
  4. Most other scientific data formats support the 64-bit integer data type, so FITS suffers by comparison. Scientific research is becoming more interdisciplinary, so it is important to be able to import data files from other fields such as space physics, planetary astronomy, and earth sciences into FITS. This is made more difficult if FITS does not support the same basic data types as the other data formats.
  5. From an abstract "data model" point of view, a primary array or image extension is identical to a table with 1 row and 1 column that contains the image as a vector. Conceptually, one should be able to convert any table vector into a FITS image array. Thus, if 64-bit integers are to be supported in FITS tables, then that data type should logically also be supported in FITS images.
  6. From an implementation point of view, once FITS I/O (Input/Output) software have been upgraded to support 64-bit integers in FITS tables, there is very little added cost to support 64-bit integer images (because tables and images are interchangeable, as described in the previous bullet).
  7. "Build it, and they will come": One should not underestimate the ingenuity of users and programmers in finding uses for new features. If 64-bit integer arrays are supported in FITS, users will discover novel applications for them that were not anticipated.