Hardware Reference
In-Depth Information
with petascale capabilities, DNS of turbulent combustion has only been able to
achieve conditions that are one generation removed from those in real devices,
which makes it essentially an exascale-class problem. Nonetheless, petascale
DNS provide a wealth of information and fundamental insight in canonical
configurations that is invaluable in developing and assessing models which
enable engineering design of combustion systems such as internal combustion
engines and gas turbine combustors.
S3D [2], developed at Sandia National Laboratories, is a massively paral-
lel DNS code used to perform simulations of gas-phase turbulent combustion
using explicit high-order finite difference numerics. The petascale simulations
performed by S3D typically generate massive amounts of data to provide suf-
ficient information spanning a broad range of spatio-temporal scales. This can
be illustrated using the recent S3D simulation of a reacting transverse jet in
a vitiated cross-flow. This configuration is relevant in gas-turbine combustors
that employ secondary fuel injection downstream of a primary combustion
zone to enable variable load operating conditions. In such a configuration, the
thermo-acoustics of the system are a serious concern and the simulation was
designed, in conjunction with experimental research at Georgia Institute of
Technology, to study the stability characteristics of the transverse jet. Topics
being investigated include (i) what are mechanisms of flame stabilization and
how are they affected by the vitiated conditions of the cross-flow, (ii) is the
transverse jet convectively unstable or globally unstable, (iii) how does the
heat release by the chemical reactions influence the stability of the jet, and
(iv) how is the mixing between the jet and cross-flow influenced by the jet
instability and how does it couple with the flame stabilization?
The simulation corresponds to a three-dimensional rectangular Cartesian
domain comprised of 4.68 billion grid points. The state vector consisted of
18 variables representing the full thermo-chemical state of a synthesis gas re-
acting system which results in approximately 84 billion degrees of freedom.
The simulation was performed on 100,000 cores of the \Edison" system at
NERSC and it periodically saves the full state (every 600 timesteps) to disk
for post-processing and analysis. The file size of each of these snapshots is 674
GB, and a total of 300 snapshots were saved making the total amount of data
generated equal to nearly 200 TB. During simulation, the I/O challenge was
to minimize the impact of storing 674-GB snapshots on the overall runtime.
However, during the post-processing and analysis, the disk storage and mem-
ory requirements are obviously very large since all 300 snapshots need to be
processed upon (not always simultaneously).
22.2 Software and Hardware
File formats used in many conventional I/O libraries such as HDF5 and
NetCDF4 require data to be written contiguously on disk. This is a natural
