Getting Started - Python and HDF5 - page 19

>>> f = h5py . File ( "name.hdf5" , driver = "core" )
You can also tell HDF5 to create an on-disk “backing store” file, to which the file image
is saved when closed:
>>> f = h5py . File ( "name.hdf5" , driver = "core" , backing_store = True )
By the way, the backing_store keyword will also tell HDF5 to load any existing image
from disk when you open the file. So if the entire file will fit in memory, you need to
read and write the image only once; things like dataset reads and writes, attribute cre‐
ation, and so on, don't take any disk I/O at all.
family driver
Sometimes it's convenient to split a file up into multiple images, all of which share a
certain maximum size. This feature was originally implemented to support filesystems
that couldn't handle file sizes above 2GB.
>>> # Split the file into 1-GB chunks
>>> f = h5py . File ( "family.hdf5" , driver = "family" , memb_size = 1024 ** 3 )
The default for memb_size is 2 31 -1, in keeping with the historical origins of the driver.
mpio driver
This driver is the heart of Parallel HDF5. It lets you access the same file from multiple
processes at the same time. You can have dozens or even hundreds of parallel computing
processes, all of which share a consistent view of a single file on disk.
Using the mpio driver correctly can be tricky. Chapter 9 covers both the details of this
driver and best practices for using HDF5 in a parallel environment.
The User Block
One interesting feature of HDF5 is that files may be preceeded by arbitrary user data.
When a file is opened, the library looks for the HDF5 header at the beginning of the
file, then 512 bytes in, then 1024, and so on in powers of 2. Such space at the beginning
of the file is called the “user block,” and you can store whatever data you want there.
The only restrictions are on the size of the block (powers of 2, and at least 512), and that
you shouldn't have the file open in HDF5 when writing to the user block. Here's an
example:
>>> f = h5py . File ( "userblock.hdf5" , "w" , userblock_size = 512 )
>>> f . userblock_size # Would be 0 if no user block present
512
>>> f . close ()
>>> with open ( "userblock.hdf5" , "rb+" ) as f : # Open as regular Python file
... f . write ( "a" * 512 )