Database Reference
In-Depth Information
allow scientists to try different mathematical algorithms by simply chang-
ing the metadata. Beyond providing information about the structure of the
data, ADIOS also has built-in support for collecting and forwarding to the
I/O subsystem key performance information, enabling dynamic feedback for
scheduling storage-related I/O, the external configuration of data collection
and storage/processing mechanisms, and value-added, additional in-flight and
o
ine/near-line processing of I/O data—for example, specifying that the data
be reduced in size (using a program that is available where the data is) before
the data is written to the disk.
ADIOS encodes data in a compact, tagged, binary format for transport.
This can either be written directly to storage or parsed for repackaging in
another format, such as HDF-5 or netCDF. The format consists of a series
of size-marked elements, each with a set of tags-values pairs to describe the
element and its data. For example, an array is represented by a tag for a name,
a tag for a data path for HDF-5 or similar purposes, and a value tag. The
value tag contains rank of the array, the dimensional magnitude of each rank,
the data type, and the block of bytes that represent the data. In the previous
example where we showed the XML data markup, the large array written
during the restarts for GTC is zion. Zion has rank
=
4, the dimensions are
(n
mi) on each process, and the block of bytes will be 4*n*4*2*mi
bytes. The remainder of this section demonstrates the utility of ADIOS for
driving future work in high-throughput data movement, by using it with two
different asynchronous data capture and transport mechanisms: the Rutgers
DART system and the Georgia Tech LIVE DataTap system. Before doing so,
we first summarize some of the basic elements of ADIOS.
ADIOS exploits modern Web technologies by using an external XML con-
figuration file to describe all of the data collections used in the code. The
file describes for each element of the collection the data types, element paths
(similar to HDF-5 paths), and dynamic and static array sizes. If the data rep-
resents a mesh, information about the mesh, as well as the global bounds of
an array, and the ghost regions
*
used in the MPI programming is encoded in
the XML structure. For each data collection, it describes the transport mech-
anism selection and parameters as well as pacing information for timing the
data transmissions. With this information, the ADIOS I/O implementation
can then control when, how, and how much data is written at a time, thereby
affording ecient overlapping with computation phases of scientific codes and
proper pacing to optimize the write performance of the storage system.
A basic attribute of ADIOS is that it de-links the direct connection be-
tween the scientific code and the manipulation of storage, which makes it
possible to add components that manipulate the I/O data outside the realm
of the supercomputer's compute nodes. For example, we can inject filters that
generate calls to visualization APIs like Visit,
11
route the data to potentially
×
4
×
2
×
*
Ghost regions are regions that overlap adjacent grid cells.
