Hardware Reference
In-Depth Information
ity of Parallel-NetCDF brings enormous computational resources to a huge
population of scientists who would rather not be bothered with the details of
MPI-IO or any other parallel I/O middleware. Furthermore, Parallel-NetCDF
provides additional capabilities that users can grow into as they become com-
fortable with their parallel computing environments.
HDF5 is similar to Parallel-NetCDF in that it is a general-purpose technol-
ogy for managing scientific and engineering data, and it provides a portable
container for distributing data. Its biggest difference, in the context of this
topic, is that it was designed from the start to perform well on high-end sys-
tems. Most notable in this regard are the ability of the HDF5 format to store
and access data objects of virtually any size eciently, and the existence of a
software layer in the HDF5 library that permits I/O drivers to be written for
new I/O modalities. The generality of HDF5 also allows applications them-
selves to include special data structures, such indexes computed by FastQuery,
that can greatly improve accessibility of data.
ADIOS has many of the same capabilities we've already seen, but empha-
sizes particularly matching the data and I/O requirements of an application
on a given system with the underlying capabilities of any of a variety of file sys-
tems. The ADIOS API hides and exploits the capabilities of POSIX, MPI-IO,
Lustre, DataSpaces, and others. ADIOS pays particular attention to scala-
bility, for instance, supporting aggregation into a single file or several files
if the single file approach fails to scale adequately. Similarly, ADIOS can im-
prove scalability in the way that it minimizes communication among processes
when handling I/O. ADIOS comes with its own format (BP), but provides file
conversion to HDF5 and netCDF.
At first glance, GLEAN appears quite similar to ADIOS and PLFS, and
the differences are definitely in the details. Like ADIOS, GLEAN adapts I/O
strategies to the underlying network topology, and makes use of the data se-
mantics of applications in order to improve data movement, but it has its own
architecture and approach. GLEAN also performs asynchronous data stag-
ing, ecient compression, and subfiling. Unlike ADIOS and PLFS, GLEAN
directly supports the HDF5 and netCDF I/O libraries.
12.3 What Is the Future of I/O Libraries?
Given their growing acceptance and institutional support, it seems likely
that at least some of the I/O libraries we see here will grow and prosper. The
success of ADIOS, PLFS, and GLEAN suggests that more packages will be-
come available that enable applications to adapt their I/O to specific architec-
tures and interconnect topologies, especially on high-end computing systems.
Their successes with specific applications will lead more and more appli-
cation developers to design the I/O in their applications to take advantage
