Databases Reference
In-Depth Information
CHAPTER 2
Getting Started
HDF5 Basics
Before we jump into Python code examples, it's useful to take a few minutes to address
how HDF5 itself is organized.
Figure 2-1
shows a cartoon of the various logical layers
involved when using HDF5. Layers shaded in blue are internal to the library itself; layers
in green represent software that uses HDF5.
Most client code, including the Python packages h5py and PyTables, uses the native
C
API
(HDF5 is itself written in C). As we saw in the introduction, the HDF5 data model
consists of three main
public abstractions
: datasets (see
Chapter 3
), groups (see
Chap‐
ter 5
), and attributes (see
Chapter 6
)in addition to a system to represent types. The C
API (and Python code on top of it) is designed to manipulate these objects.
HDF5 uses a variety of
internal data structures
to represent groups, datasets, and at‐
tributes. For example, groups have their entries indexed using structures called “B-trees,”
which make retrieving and creating group members very fast, even when hundreds of
thousands of objects are stored in a group (see
“How Groups Are Actually Stored” on
page 65
). You'll generally only care about these data structures when it comes to perfor‐
mance considerations. For example, when using chunked storage (see
Chapter 4
), it's
important to understand how data is actually organized on disk.
The next two layers have to do with how your data makes its way onto disk. HDF5
objects all live in a 1D logical address space, like in a regular file. However, there's an
extra layer between this space and the actual arrangement of bytes on disk. HDF5
drivers
take care of the mechanics of writing to disk, and in the process can do some amazing
things.
