Hardware Reference
In-Depth Information
which share a given I/O node will likely not be sequential and as such, writing
a shared file can become problematic if I/O requests are smaller than a file
system block size. If the same block is needed by more than one I/O node,
then those updates will be serialized by GPFS.
The other common problem faced by new users is that of \rank zero" I/O.
This I/O model either reads or writes the entire file from a single MPI rank.
The rank zero model will work for small files, such as input configuration,
but for simulation data, this model becomes bottlenecked because it will only
utilize a single I/O node. That will limit the output rate to around 300 MB/s.
4.6 Future I/O Plans at ALCF
With the second generation of compute and storage systems at the ALCF,
staff feel that their project-based file system strategy has been successful and
useful to their users. They will continue to improve this process by adding
quota enforcement to the file system and tools for users to better manage
data usage.
While ALCF users have generally provided positive feedback on the single
file system solution, it does have one glaring issue: it has a single point of
failure. If the file system goes down, the entire facility is down. In order to
address this issue on Mira, ALCF is working with IBM to use the Active
File Management (AFM) feature of GPFS to achieve the desired benefits, like
a single namespace with no additional file management for the users, while
eliminating the single point of failure. To accomplish this, ALCF purchased
an additional six DDN SFA12Ke storage arrays, which are identical to the
ones already in place. These were brought up as a separate 7-PB file system
and achieved 90 GB/s with IOR. A total of 30 IBM GPFS Storage Server
(GSS) Model 24 units (where 24 means 2 servers and 4 JBOD units) were
also purchased and a third 13-PB file system was created across them. GSS
takes IBM x3650 servers and NetApp Engenio JBOD drawers, and rather
than use the NetApp RAID heads, uses the GPFS Native RAID (software
RAID) implementation. The GSS file system achieved 400 GB/s with IOR.
As shown in Figure 4.6, the design calls for all users to write to the GSS cache,
and then have AFM transparently migrate the files out of the cache to their
home directory. This configuration increases the peak performance available,
maintains a single namespace, requires no effort on the part of the users, and
removes the single point of failure.
In the near future, ALCF wishes to extend this concept to tape as well.
They already employ HPSS for archival tape storage. There is a software
tool called the GPFS/HPSS Interface (GHI), which allows HPSS to act as a
hierarchical storage manager for GPFS. With this system in place, the file life
cycle would follow as:
