Hardware Reference
In-Depth Information
organization for serial programs where the entire dataset is contained within
a single process. However, for parallel programs, datasets (e.g., a multi-
dimensional array) are likely scattered across multiple processors. In order
to make data logically contiguous on disk, extensive communication must oc-
cur among MPI processes either to aggregate data chunks on a single process
or to exchange write offsets to enable collective, non-overlapping writes. As
scientic codes are scaled up, this involves many \all-to-one" or \all-to-all"
collectives and will be a significant performance bottleneck. Meanwhile, scien-
tific simulations often do calculations and I/O in an iterative manner (i.e., for
timesteping). For a contiguous file format, appending to an existing dataset
can be a very costly operation. It requires heavy data shuing on disk and
the rewriting of substantial file metadata.
22.2.1 ADIOS-BP
Keeping these limitations in mind, we designed ADIOS-BP, which adopts
an alternative approach, i.e., a non-contiguous, log-based file format. Choosing
the format to be non-contiguous allows BP to scale extremely well to hundreds
of thousands of cores. There are two main factors behind the performance
advantage that this format brings. First, the non-contiguity allows each MPI
processor to perform I/O in a more independent fashion, significantly reducing
(if not eliminating) collective communications. Each processor outputs its local
buffer, along with header information such as indexes and data dimensionality,
independently to a file. It completely eliminates the MPI communication step
that two-phase I/O involves to make data appear contiguous on disk. In a
non-contiguous le format like ADIOS-BP, \all-to-one" data shuing is not
required. Additionally, non-contiguity makes file system lock contention much
easier to resolve, particularly when the subfile technique is applied. The key
elements are briefly explained in the ADIOS-BP file format:
Process Group: One of the major concepts in ADIOS-BP format is \pro-
cess group" or PG. The ADIOS-BP le format consists of a series of PG
entries and the ADIOS-BP file footer, see Figure 22.2. Each process
group is the entire self-contained output from a single process and is
written out independently into a contiguous disk space.
Footer: One known limitation of the NetCDF4 format is that the file
contents are stored in a header that is exactly big enough for the infor-
mation provided at file creation. Any changes to the length of that data
will require moving data. To avoid this cost, a foot index is employed
instead. The version identifier and the offset are placed at the beginning
of the index as the last few bytes of the file, making it simple to find
the index information and to add new and different data to the files
without aecting any data already written. The variable's index table is
composed of the total count of variables in the ADIOS-BP file, the size
of the variable's index table, and a list of variable records. Each record
