Hardware Reference
In-Depth Information
LANL applications use both the N processes writing to N files (N-to-N)
and N processes writing to 1 (N-to-1) file modes of I/O. However, the bulk
of the cycles used by LANL users utilize the N-to-1 file mode. The primary
applications at LANL are able to use POSIX and MPI-IO as I/O methods and
one code utilizes HDF5 heavily. The bulk of the I/O is used for checkpoint
restart but some I/O is used for data analysis and visualization.
Users utilize the scratch file system as much as possible, but given that
this scratch file system is not backed up or automatically archived, users must
decide what data to save to the parallel archive system. Over the past 25
years, LANL has seen an increase in archive storage growth by about three
memories/month; although recently, the size of the working sets users want
to save to the parallel archive has been stretching the ability of the parallel-
tape-based systems, such as HPSS and IBM GPFS/TSM.
6.3.2 I/O Patterns and the Quest for Performance without
Giving Up
The applications that use the most resources at LANL utilize the N-to-
1 file I/O pattern. In fact, it is N-to-1 strided, making a small unaligned
I/O access pattern for the parallel file systems. The applications prefer this
method of I/O because it allows them to easily manage restarting to a different
number of processors, which is complicated as the code is an Automated Mesh
Refinement [3] code. The users also find it easier to manage the one-file-per-
run as well, for management and archive reasons.
One problem, however, is that all three top parallel file systems do not
perform N-to-1 using a strided pattern well, even though N-to-1 is really an
excellent way for the users to work. LANL staff worked hard to use vari-
ous forms of collective buffering [13], persistent file domains [8], and other
techniques to help maintain N-to-1 performance, as opposed to switching to
N-to-N for I/O. Luckily, LANL's HPC center was able to keep the N-to-1
strided model for users, and today they see the pay off since N-to-N is not
user friendly when hundreds of thousands of cores are each producing a file
every few hours.
6.3.3 Defeating N-to-1 Strided
After trying to get collective buffering and other alignment techniques to
work well for years, LANL and Carnegie Mellon University (CMU) decided
to try parallel log structured storage. An MPI-IO ADIO (Abstract Device
Interface for I/O) [12] implementation was contemplated, but LANL pursued
a more ambitious parallel log-structured file system (PLFS) [2] layer because
it would work with many I/O application performing interfaces (APIs).
The PLFS turns N-to-1 file trac into N-to-N files as depicted in Fig-
ure 6.4, which yields good performance without the user having to deal with
