Hardware Reference
In-Depth Information
(e.g., the DIMES and FlexPath methods of ADIOS). The DataSpaces model
by Docan et al. [6], however, provides a virtual shared multi-dimensional space
using a set of separate nodes as a \staging area." Thus multiple, parallel appli-
cations can simultaneously read a multi-dimensional array, with an arbitrary
decomposition. More importantly for interactive visualizations, DataSpaces
can hold as many output steps as the allocated memory can hold. Moreover,
DataSpaces provides fault isolation for the application. Failures downstream
in a process pipeline do not propagate to the application. More details on us-
ing ADIOS and DataSpaces for in-transit visualization can be found in the \In
Situ Processing" chapter of the topic on High Performance Visualization [2].
17.3.5 Data Reduction
Data reduction techniques with ADIOS have shown to be effective in re-
ducing the bottleneck on I/O during simulation writes. These techniques fit
well with the minimal communication principle employed by ADIOS where
each process handles its own I/O. By applying compression routines locally on
every process, encoding costs are effectively minimized. This also enables com-
pression techniques to take advantage of similarity in data values within each
process, typically seen with spatio-temporal scientific datasets. For example,
ISOBAR [12], an in situ lossless compression routine specific to scientific data,
demonstrated up to a 46% reduction in storage on datasets from simulations
spanning various domains such as combustion (S3D), plasma (XGC1, GTS)
and astrophysics (FLASH). This technique coupled with ADIOS and with the
addition of interleaving allowed throughput gains proportional to the degree
of data reduction.
While checkpoint/restart data, such as particle data, must be compressed
losslessly, datasets that are used for analysis and visualization, such as field
data, can be compressed in a lossy fashion. Unlike lossless compression, lossy
compression techniques, such as wavelets-based compression (i.e., ISABELA
[8], etc.) provide a multi-fold reduction in storage sizes, trading precision for
storage reduction. Depending on the sensitivity of the analysis routines to er-
rors introduced by the compression process, the end users can change the level
of accuracy desired in the configuration file used by ADIOS. The configura-
tion file can also instruct ADIOS to employ different compression routines for
different variables and output groups (checkpoint/restart or analysis) without
having to change the application code.
17.3.6 Deployment
ADIOS is an open-source software with a BSD license. It has been installed
as central software and is supported by the Oak Ridge Leadership Facility, by
the iVEC organization in Western Australia for the purpose of supporting the
Square-Kilometer Array project, by the High Level Support Team (HLST)
supporting EFDA (European Fusion Development Agreement) sites, and by
