Hardware Reference
In-Depth Information
at Urbana{Champaign set out to create an architecture-independent software
library and file format to address the need to move scientific data among
the many different computing platforms at NCSA. Additional goals for the
project included the ability to store and access large objects eciently, the
ability to store many objects of different types together in one container, the
ability to grow the format to accommodate new types of objects and object
metadata, and the ability to access the stored data from both C and Fortran
applications.
Originally dubbed AEHOO (All Encompassing Hierarchical Object Ori-
ented format), the new software and file format was ultimately called Hier-
archical Data Format (HDF), and was developed as an open-source product
and distributed free of charge under a University of Illinois license. The design
of HDF combined ideas from a number of different formats including TIFF,
CGM, FITS, and the Macintosh PICT format.
In 1996 the HDF group began a successful collaboration with the Depart-
ment of Energy's (DOE) Advanced Simulation and Computing (ASC) pro-
gram. The goal of ASC was to increase the computing power of DOE systems
at Lawrence Livermore, Los Alamos, and Sandia National Laboratories by
several orders of magnitude. This boost in computational power was clearly
going to require an associated scale-up in data management capabilities|data
files would be very large, written in parallel on massively parallel systems, and
the data itself would be more complex than ever.
A Data Models and Formats (DMF) group, drawn from the three DOE
laboratories and NCSA, set about to create a format that would address ASC's
needs. HDF had many of the required features, but it didn't scale to suciently
large files, and could not easily support parallel I/O. Like the GFTF before it,
the DMF borrowed ideas and lessons learned from other formats, in particular
from HDF and a format from Livermore called Array I/O (AIO), to define a
new format for the project. Initially called \Big HDF," the resulting format
was nally named \HDF5" because the latest release of the original HDF was
HDF version 4.0.
The DOE laboratories and NCSA, with additional support from NASA,
jointly developed and released the first version of HDF5 in 1998. HDF5 con-
tinues to enjoy a distinguished status as a scientific data format and library,
with successful collaborations with NASA and DOE continuing to the present
day.
16.3 Design and Architecture
HDF5 is designed at three levels: a data model, a software library, and a
file format. The HDF5 data model consists of abstract classes, such as files,
datasets, groups, datatypes, and dataspaces, that application developers use
