Hardware Reference
In-Depth Information
Increasing FIle oFFset
(a) Decomposed mesh
(b)
File
mapping
FIGURE 30.1 (See color insert): Domain decomposition resulting in highly
non-contiguous access pattern. Color indicates node access.
HPC applications also tend to differ in how files are accessed. Often, a
computational problem is partitioned between a set of nodes working on a
single problem. For example, in the case of a weather simulation application,
the three-dimensional region for which the weather is computed is sliced into
multiple pieces and distributed among the nodes. When a part of a higher-
dimensional region gets mapped to a single-dimensional file (when each node
writes its portion of the simulated region to disk) a highly non-contiguous
access pattern emerges. This is demonstrated in Figure 30.1, which shows a
detailed 3D mesh (a), where color indicates the partitioning across nodes, and
the resulting mapping to a one-dimensional file (b).
Unfortunately, the standard POSIX functions do not offer a way of ex-
pressing an I/O operation that is non-contiguous in the file (non-contiguous
in memory is supported through the
readv
family of functions), forcing the
application programmer to issue many small contiguous I/O requests instead,
leading to severely reduced I/O performance. The
readx
and
writex
func-
tions, proposed by the POSIX HPC extensions, aim to resolve this by offering
functionality to describe an I/O transfer which can be non-contiguous in mem-
ory as well in the file.
The POSIX HPC I/O extensions also enable relaxing the rather strict
POSIX I/O consistency semantics. For example, a new open flag is proposed,
OLAZY
. The effect of this flag is to reduce the consistency scope of data and
metadata to the scope of the process or local node, as opposed to all nodes
mounting the same file system. When system-wide consistency is required, an
application can call
lazyiopropagate
to propagate modifications to other
processes or node, and
lazyiosynchronize
to invalidate any locally cached
data or metadata.
