Hardware Reference
In-Depth Information
the root group of the le (referred to as \/dset"), and writes all the elements
of the array out from a memory buffer. The HDF5 programming API and its
usage is extensively documented online with example code, tutorials, a user's
guide, and API reference manual [14].
16.3.3 The HDF5 File Format
Based on many years of experience developing the original HDF package
(now called \HDF4" [12]) and in broadly surveying many other le formats
and data storage mechanisms, the HDF5 file format is designed to be portable,
resilient to data corruption, and future-proof through \micro-versioning."
Micro-versioning within an HDF5 file allows individual aspects of each ob-
ject to be stored with a different version. This capability allows the format of
individual components of objects to be incrementally extended and improved
as needed, without version changes for the entire file format.
Files created by the HDF5 library are always backward compatible and are
as forward compatible as possible. Backward compatibility indicates that old
files continue to be able to be accessed with newer versions of the HDF5 library.
Forward compatibility allows files created by later versions of the library to be
read by earlier versions, to the extent possible. The HDF5 library implements
this capability by storing each object in a container with the oldest micro-
version of the object's components that can possibly represent the information
and features requested for the object. This allows the greatest range of older
released software to read files produced by newer libraries.
In a very strong sense, HDF5 files implement a file system within a con-
tainer, with groups analogous to directories and datasets similar to (struc-
tured) files. As such, there are many pieces of infrastructure within an HDF5
container, such as B-trees, heaps, etc. All aspects of the HDF5 file format
are described in the format specification [15], which has allowed third-party
implementations of HDF5 file readers, proving the ability of HDF5 files to
outlast even the current library implementation.
16.4 Usage and Applications
HDF5 is used across hundreds of science, engineering, and business do-
mains and by thousands of applications within those domains, from high-
energy physics, climate modeling, and financial market tracking to airplane
flight test data recording and medical record storage. The following vignettes
highlight some of the high performance computing applications of HDF5 that
are most relevant to this text.
