Hardware Reference
In-Depth Information
All results were collected on NERSC's Cray XE6 system, Hopper, which
has 6,384 twenty-four-core nodes with 32 GB of memory each. It features a
Gemini interconnect with a 3D torus topology, and a Lustre parallel file system
with 156 Object Storage Targets (OSTs) and an advertised peak bandwidth
of 35 GB/s. The OSTs are backed by 26 Object Storage Servers (OSSs),
which share 13 LSI 7900 disk controllers. Lustre router nodes in the Gemini
interconnect forward I/O requests from the Hopper compute nodes to the
OSSs through a separate QDR InfiniBand interconnect.
Iota's overhead, as a percentage of overall runtime, remained low even as
the runs scaled up to 110,052 MPI tasks (see Figure 28.1). We expect that
tracing of real applications should incur even less overhead, since our test
applications are all I/O benchmarks that do nothing but issue I/O calls.
We ran Iota in gather mode (instead of flushing mode) to validate the
feasibility of gathering the entire trace to the root task for output to a single
file. The results showed that gathering the trace accounted for very little of the
overhead (see Figure 28.1). After further investigation, we found considerable
variation in the overhead for calls into the Lustre API to obtain striping
information. This is likely due to the design of the Lustre file system, with its
centralized metadata server.
In all cases, no MPI tasks used more than the initial 2 MB allocated to
buffer the string data for the trace entries. In
MPIFinalize
, the root task
allocated a buffer as large as the final trace output, which ranged from 1.8
MB to 139 MB across the test runs. In practice, we expect that the application
has freed its allocated memory before finalizing, so that additional memory is
available for Iota at this point.
Figures 28.2 and 28.3 show examples of the plots we can generate when
complete traces are available for post-processing.
FIGURE 28.1: Overhead of the Iota library, tested with three I/O benchmarks.
