Hardware Reference
In-Depth Information
5. Application-controlled throttling of I/O is easily supported in a MIF-
IO setting because the number of concurrent operations is explicitly
controlled. This can help to avoid overloading the underlying I/O sub-
systems.
6. MIF-IO is consistent with the way leading-edge commercial \big data"
I/O in map-reduce operations is handled. Datasets are broken into pieces
and stored in the file system as a collection of shards and different num-
bers of parallel tasks can process different numbers of shards.
With minor variations, the process by which ALE3D writes plot files is
completely analogous to restart files. However, the application will often en-
able some \in-transit" transformations on data destined for plot les that
it would otherwise not perform for restart files. This can include transfor-
mations from double to single precision, the addition of ghost zone layers
enveloping domains, the inclusion of domain spatial and variable extents as
well as both lossless and lossy compression. In earlier systems where compu-
tations were performed on Crays and visualizations on SGIs, Silo would also
perform transformations from Cray to SGI floating point representations.
Nonetheless, both restart and plot files, because they are Silo files, are still
visualizable by VisIt. Plot files are designed to be a little more convenient
and optimized for visualization in VisIt, while restart files are optimized for
restarting ALE3D.
When VisIt [6] reads a Silo restart or plot file, it starts by opening the
master le. From that le, VisIt's metadata server determines all of the objects
in the le, their names and types, and then populates VisIt's GUI menus based
upon what it finds in the files. There are a number of interesting aspects to how
VisIt manages data distributed across multiple files. What is most relevant
here is that VisIt essentially piggy-backs its parallel processing paradigm on
top of the parallel decomposition already provided by the data producer.
Like ALE3D, VisIt can process multiple domains on each MPI task. When
the user starts VisIt, she specifies a suitable number of MPI tasks to use. VisIt
then automatically assigns domains to tasks (it has a number of different al-
gorithms for doing this depending on various factors). Each task then uses
information from the master file to determine which file(s) contain the do-
mains it needs and then opens those files and reads the relevant domains from
them.
If the data producer bothered to write per-domain spatial and variable
extents, VisIt can use such information to accelerate various operations. For
example, to display a slice of a mesh, VisIt can exclude from processing any
domain whose spatial extents do not intersect the slice. Likewise, when dis-
playing iso-surfaces, VisIt can exclude from consideration any domain whose
variable extents do not contain the iso-values of interest.
