Hardware Reference
In-Depth Information
topologies including the 5D torus (IBM BG/Q), dragonfly (Cray Cascade),
and 6D torus (K-machine). Additionally, our community is witnessing sys-
tems designed to include burst buffer and dedicated analysis nodes. Parallel
file systems deployed at supercomputing centers tend to have diverse perfor-
mance characteristics for I/O patterns including a single shared file versus a file
per process, due to the design and implementation of their underlying meta-
data and lock management mechanism. Applications need to deal with these
factors, among others, in order to scale their I/O performance. To overcome
these bottlenecks and help increase the scientific output of leadership facilities,
GLEAN provides a topology-aware mechanism for improved data movement,
compression, subfiling, and staging for accelerating I/O, interfacing to running
simulations for co-analysis, and/or an interface for in situ analysis requiring
little or no modification to the existing application code base.
18.2 Design and Architecture
has been explicitly designed to improve application performance and pro-
ductivity. GLEAN users can move data out of the simulation with minimal
overhead using a clean interface. The GLEAN design aims to:
exploit the underlying network topology to speed data motion of of the
supercomputer;
provide asynchronous data I/O via staging nodes;
develop scalable mechanisms for collective I/O by reducing synchroniza-
tion requirements;
mitigate variability in I/O performance of shared file systems using stag-
ing;
leverage data semantics of applications;
enable simulation-time data analysis, transformation, and reduction;
provide non-intrusive integration with existing applications; and
provide transparent integration with native application data formats.
Figure 18.1 provides an overview of the GLEAN infrastructure and com-
pares the traditional mechanism used for I/O with GLEAN. The simulation
running on the compute nodes may invoke GLEAN directly or transparently
through a standard I/O library such as Parallel-netCDF [8] and HDF5 [6]. The
data is moved out either directly to storage or to dedicated analysis/staging
