Hardware Reference
In-Depth Information
common bottlenecks that would hamper that performance. The many pro-
cesses writing to different offsets in a file or to different files avoid each other
on a parallel file system as much as possible. In most cases, each process
attempts to write to a single stripe target to avoid the metadata server over-
head of spanning storage targets. The read performance of this file format was
shown to be generally advantageous by Lofstead et al. [10].
17.3 Deployment, Usage, and Applications
17.3.1 Checkpoint/Restart
Most applications face the I/O bottleneck when they are running at scale
and need to perform checkpointing. Writing checkpoint data represents two
classes of I/O challenges: write (1) large or (2) small amount of data, stored in
many variables, from every process of the application. In both cases, buffering
helps to decrease the amount of individual write operations, thus avoiding I/O
latency and utilizing the available I/O bandwidth. In both cases, aggregation
is required to decrease the number of entities hitting on the file system at
the same time. ADIOS has consistently improved all large-scale applications'
checkpoint writing (and reading) performance by 10{100x and thus allowed
for writing checkpoint data of the size of multiple terabytes at a time with
acceptable I/O overhead, as reported in Lofstead et al. [10, 11].
An application, however, does not need to run on tens of thousands of cores
to experience I/O bottlenecks. For example, one application is Fine/Turbo,
a Computational Fluid Dynamics (CFD) code from Numeca International,
that is used in industrial engineering. This CFD simulation simulates billions
of cells to get new insights in turbulent behaviors. The irregular topology
of a specific simulation is partitioned into small regular 3D blocks. Its load-
balancing algorithm divides up the blocks and the variables on each block
among the processes so that different variables in a block have different sets
of processes assigned to them. The original sequential I/O naturally could not
scale up to thousands of cores. Collecting all data on a single core led to high
communication overhead. Attempts to write in parallel, while retaining each
variable of each block organized on the disk as a contiguous block of data, led
again to too much communication between processes.
ADIOS provided the optimal solution to the writing problem. Each process
dealt with its own I/O, thus completely avoiding cross-communication among
the processes. The data on each process was buffered locally. Then, because
of the ADIOS-BP file format, data was written into a separate segment of the
output file, that is, with one write request with a single seek in the output. The
original sequential I/O took 2000 seconds to complete writing (a smallish) 18
GB of output of a 500-million-cell simulation running on 1280 cores. ADIOS,
