Hardware Reference
In-Depth Information
mandated for the last two archives of the Climate Model Intercomparison
Project, associated with IPCC assessment reports. More than 100 open-source
software packages provide netCDF access. NetCDF is also used in other dis-
ciplines, including chromatography, neuro-imaging, molecular dynamics, and
fusion research. NetCDF data access is included with many commercial soft-
ware packages, such as ESRI ArcGIS, MATLAB, and IDL.
16.4.2
H5hut
16.4.2.1
Design
Particle-based simulations running on large high performance computing
systems over many timesteps can generate an enormous amount of particle-
and field-based data for post-processing and analysis. Achieving high perfor-
mance I/O for this data, effectively managing it on disk, and accessing it from
analysis and visualization tools can be challenging, especially for domain sci-
entists who do not have I/O and data management expertise. The H5hut [7]
library is an implementation of several data models for particle-based simula-
tions that encapsulates the complexity of parallel HDF5 and is simple to use,
yet does not compromise performance.
H5hut is a veneer API for HDF5: H5hut files are also valid HDF5 files
and are compatible with other HDF5-based interfaces and tools. For example,
the h5dump tool that comes with HDF5 can export H5hut files to ASCII or
XML for additional portability. H5hut also includes tools to convert H5hut
data to the Visualization ToolKit (VTK) format and to generate scripts for
the GNUplot data plotting tool.
H5hut is tuned for writing collectively from all processors to a single,
shared file. Although collective I/O performance is typically (but not always)
lower than that of file-per-processor, having a shared file simplifies scien-
tific workflows in which simulation data needs to be analyzed or visualized.
In this scenario, the file-per-processor approach leads to data management
headaches because large collections of files are unwieldy to manage from a
file system standpoint. Often a post-processing step is necessary to refactor
file-per-processor data into a format that is readable by the analysis tool. In
contrast, H5hut files can be directly loaded in parallel by visualization tools
like VisIt and ParaView.
16.4.2.2
Applications
OPAL (Object Oriented Parallel Accelerator Library) is a tool for
simulating charged-particle optics in large accelerator structures and beam
lines [1]. OPAL is based on IP
2
L [2] and provides a data parallel approach
for particles, fields, and associated operators. Production runs of OPAL codes
use several thousand cores, on the order of 10
9
simulation particles, and mesh
resolutions on the order of 1024
3
, which can be saved for post-run analysis
using H5hut.
