Hardware Reference
In-Depth Information
code was interfaced with a running instance of ParaView on an associated
visualization cluster and the visualization produced while the simulation was
running (see Section 18.3.3).
Similarly, the Adaptable I/O System (ADIOS) library has been extended
to allow for data to be \staged" to a set of nodes where processing can be
performed before sending the final dataset to storage [1, 9]. In such a setup,
the I/O library provides the application with a mechanism to move the data
to storage by way of a staging node where the I/O library can invoke user-
supplied analysis or data transformation (such as converting the data to a
specific format like HDF5) libraries. A similar concept was introduced with
ActiveSpaces [3], which provides a mechanism that allows for data to be pro-
cessed through a staging node on the way to persistent storage. Additional
details on ADIOS are discussed in Chapter 17.
One final prototype of in-transit analysis is a collaboration between Los
Alamos National Laboratory and EMC to develop a prototype system con-
sisting of a burst buffer connected between a cluster and a Lustre parallel
file system [2]. In this demonstration, a simulation of the fluid dynamics of
a wind turbine model was run such that as timesteps were dumped to the
burst buffer, a ParaView instance was run to produce a visualization at each
timestep. Comparing the runtime of the tests, it was clear that by overlapping
the visualization process with the simulation execution, time-to-results was re-
duced. This reduction was approximately 33% of of the runtime when using
burst buffers as compared to the same run without the benefit of their use.
23.4 Conclusion
In-transit processing of data is a natural extension of placing burst buffers
into HPC systems. With data being held close to a processing element in fast
storage (flash in this case), the ability to analyze the data before it is sent to
slower rotating storage provides a significant potential for decreased time to
results for users. The prototype systems discussed in this chapter are examples
of some of the initial work conducted to explore this potential and develop
effective systems to leverage in-transit analysis in HPC systems. As no current
HPC system has a burst buffer in production use, this technique of in-transit
processing will undoubtedly evolve as technology changes in the years ahead.
Bibliography
[1] Hasan Abbasi, J. Lofstead, Fang Zheng, K. Schwan, M. Wolf, and
S. Klasky. Extending I/O through High Performance Data Services. In
