Hardware Reference
In-Depth Information
20.4.1 Data Model and Versioning
One of the biggest strengths of HDF5 is that the data model lies in the file.
What is more important, there is forward and backward compatibility with
files generated by the library. This allows a very flexible view of the data when
considering parallel access to the file. As a rule of thumb, programs should
use the HDF5 format to store data that other programs should read. The
common data model is stored in the file and it is easy to explore and consume.
For simulation data, this is crucial as it allows different access patterns to be
implemented. Each program can access the same data without sharing the
same business. Furthermore, the same program between versions can freely
change its business and still be able to access any file version. In case data need
not be shared between programs, a process may store proprietary data in the
same file alongside shared data. Or, if it needs to have more granular access to
its own data, it can still use the HDF5 format. This flexibility accommodates
a broad array of use cases and serves multiprocess access and versioning very
well.
20.4.2 VFL and Filters
The HDF5 library leaves enough open ends for extending it and more
can be devised, as will be shown later. Two of the most interesting open
ends are the Virtual File Layer (VFL) and the filters. Exploiting VFL, one
may implement different modifiers and storage support drivers for data. For
example, a terminal VFL driver that stores HDF5 objects in RAM can be
used for platform-independent large data attachments in combination with
Inter Process Communication systems. Filters can also be used for encryption,
compression and other data manipulations. Another open-ended facility of
HDF5 is the definition of new data types and conversion between data types.
This feature allows for optimal performance when storing data using different
precision than the one allowed by the memory data type.
20.4.3 Encryption
The openness of HDF5 is its blessing and its curse. For commercial appli-
cations, data encryption is often imperative. However, HDF5 allows anyone
with access to a computer and HDFView to look at data, manipulate it, and
export it. This is a major shortcoming and software vendors need to devise
workarounds to restrict access to their data.
20.4.4 Robustness
HDF5 really shines when everything goes well. However, more often than
not, when something goes wrong, HDF5 fails in different ways, from making
files inaccessible to crashing processes. In most cases, the underlying reasons
