Database Reference
In-Depth Information
process a large number of different formats. Many of these formats are
built atop underlying I/O libraries, like HDF5 and netCDF. Even with well-
established I/O libraries, which implement an on-disk data format for storing
and retrieving arrays of data, there exist several issues that perennially affect
producers and consumers of data. One is the fact that the APIs for these I/O
libraries can be complex: They provide a great deal of functionality, which
is exposed through an API. Another is that it is possible to “misuse” the
I/O library in a way that will result in less-than-optimal I/O performance.
Yet another is that semantic conventions are not consistent across and within
disciplines. The existence of well-established I/O libraries is of huge benefit to
developers, for these technologies accelerate development by not reinventing
the wheel. However, they can be complex to use, and they don't solve the
semantic gap problem that exists between producers and consumers of data.
This section addresses these topics from a visualization-centric perspective.
In our work, we often are the ones who end up having to reconcile semantic as
well as format discrepancies between simulation, experiment, and visualization
technologies. As it does not seem practical for a panacea solution that will
solve these problems for all disciplines, we have adopted a bottom-up approach
that focuses on addressing these problems for communities and for specific
new capabilities at the crossover point between the fields of visualization and
scientific data management.
The first subsection below presents H5Part, which is a high-level API that
provides a solution to the semantic gap for use in computational accelerator
modeling. Concurrently, H5Part (which stands for HDF5 format for particle
data) is engineered to provide good I/O performance in a way that is rea-
sonably “immune to misuse.” It encapsulates the complexity of an underlying
I/O library, thereby providing advanced data I/O capabilities to both simula-
tion and visualization developers. The relative simplicity of its API lowers the
developer cost of taking advantage of such technology. Next, we present an-
other high-level API aimed at encapsulating both data I/O and index/query
capabilities. Again, our motivation is to encapsulate the complexity of both
of these technologies. Finally, we present some examples of how these tech-
nologies are used in practice.
9.4.1
H5Part
9.4.1.1
Motivation
Modern science—both computational and experimental—is typically per-
formed by a number of researchers from different organizations who collec-
tively collaborate on a challenging research problem. In the case of particle
accelerator modeling, different groups collaborate on different aspects of mod-
eling the entire beamline. One group works on modeling injection of particles
into the beam, another works on modeling magnetic confinement and beam
focusing along the beamline, while yet another works on modeling the impact
