Hardware Reference
In-Depth Information
Chapter 15
Parallel-NetCDF
Rob Latham
Argonne National Laboratory
15.1
Motivation :::::::::::::::::::::::::::::::::::::::::::::::::::::::: 177
15.2
History and Background ::::::::::::::::::::::::::::::::::::::::: 179
15.3
Design and Architecture ::::::::::::::::::::::::::::::::::::::::: 179
15.4
Deployment and Usage ::::::::::::::::::::::::::::::::::::::::::: 180
15.5
Additional Features :::::::::::::::::::::::::::::::::::::::::::::: 181
15.6
Conclusion :::::::::::::::::::::::::::::::::::::::::::::::::::::::: 182
15.7
Additional Resources ::::::::::::::::::::::::::::::::::::::::::::: 183
Bibliography :::::::::::::::::::::::::::::::::::::::::::::::::::::: 184
15.1 Motivation
Parallel-NetCDF (often abbreviated as simply \pNetCDF"), is a high-level
parallel I/O library providing a portable, self-describing file format for the
storage of multi-dimensional arrays of typed data. Parallel-NetCDF provides
an application-oriented interface on top of the more general, lower-level MPI-
I/O interface, while maintaining the standard netCDF file format.
Computational scientists, in addition to mastering their scientific domain,
also have the challenge of mastering the computer systems upon which their
simulations run. To help hide the hardware details of these machines, scien-
tists can use compilers, programming languages, and math kernel libraries. In
order to manage the tens of thousands of nodes and millions of processing ele-
ments, scientists can rely on communication middleware such as MPI. In much
the same way, scientists have at their disposal high-level I/O libraries. These
I/O libraries sit atop an I/O middleware (such as MPI-IO, discussed in Chap-
ter 13), packaging and presenting some of the complexity of the lower-level,
general purpose middleware into a format more appropriate for a scientific pro-
grammer. Figure 15.1 depicts this I/O software stack, and Parallel-NetCDF's
role in it.
The \data problem" facing computational scientists consists of three main
challenges. First, the simulations of physical phenomena improve their fidelity
with each new generation of hardware. As the gap between what computer
177
