Hardware Reference
In-Depth Information
VPIC is a computationally intensive application, and it is impractical to run
the entire code repeatedly, and at scale for tuning. Instead, the study devel-
oped a simplified parallel I/O kernel, called VPIC-IOBench. The kernel uses
the same H5Part calls shown in Section 19.5 for writing VPIC particle data.
VPIC-IOBench disables the simulation component of the VPIC code and uses
random data. The kernel contains the full data volume generated by the code
with a slightly simplified pattern. In VPIC-IOBench, each MPI process writes
an equal number of particles to a shared file, whereas in a VPIC simulation,
each MPI domain writes a slightly varying (up to 15%) number of particles.
The amount of data the kernel writes is proportional to the number of MPI
processes.
19.6.2 HDF5 Tuning
When a HDF5 file is closed, the
H5Fclose
function ensures that the size
of the file matches its allocated size. HDF5 tracks the size of the file for two
reasons: to detect external file modification/corruption (i.e., from something
other than the HDF5 library modifying/corrupting the file), and to allocate
space within the le for changes to the le's structure. HDF5 currently veri-
es the le's size by truncating the le (using a POSIX or MPI operation) to
the size of the allocated space within the file. This truncate-based verification
step initiates several metadata server operations that degrade performance
signicantly. Disabling the le's size verication in some cases will likely make
the file unreadable. For example, if the truncate operation during
H5Fclose
would actually extend the file (instead of truncating it), but does not oc-
cur (because of disabling truncate), data will be double allocated in the file,
causing corruption later. If the truncate was actually truncating the file (in-
stead of extending it), it is less of a problem|space in the le will be leaked,
but the le will still be readable. The le's size verication was disabled for
performance.
In all the datasets, disabling truncate did not affect the ability to read the
files. The HDF5 library source code (version 1.8.8) was modified to disable
the file size verification process, which improved performance of writing files
by a factor 3{5X. The impact of disabling the le size verication is shown
in Figure 19.3. The plot shows improvement of the I/O bandwidth of VPIC-
IOBench over a range of MPI processes counts, where each MPI process writes
32 MB of data. A better alternative to simply disabling the truncate call is to
track the valid section of the file by adding more metadata to the file format.
This optimization will soon be a part of upcoming HDF5 release allowing all
parallel HDF5 applications to more productively use HDF5 for I/O.
19.6.3 Tuning Lustre File System and MPI-I/O Parameters
To tune MPI-I/O and Lustre layers, the study extracted VPIC's I/O kernel
into a benchmark, named VPIC-IOBench. During the study, a series of tests
