Hardware Reference
In-Depth Information
FIGURE 14.3: By using the PLFS middleware layer, the illusion of a single
file is preserved in a manner completely transparent to the application and
the user. Physically however, PLFS transforms the I/O to leverage both the
global visibility of the parallel file system to store its metadata as well as
the faster performance of the storage in the burst buffers to store user data.
Later, the user data is migrated to the parallel file system. This figure shows
an example burst buffer architecture where a simulation application sends
simulation data from compute nodes to burst buffer nodes that have been
augmented with GPUs to allow in-transit analysis by a visualization program
(see Chapter 23). The data will be later migrated to a Lustre file system
running on VNX storage. [Image courtesy of John Bent (EMC).]
14.4 Conclusion
Through a variety of configurations and use cases, PLFS is a dramatic ex-
ample of the power of software-defined storage. An underlying storage system
can be configured to work well with well-arranged streams of data. The PLFS
software is then layered around that storage system and can be configured to
export it for a variety of different workloads, such as shared file, small file,
and flat file, for better metadata load balancing. Additionally, PLFS can en-
able parallel I/O using storage systems not defined for parallel I/O. Finally,
PLFS can create a single virtual file system from a collection of multiple stor-
age systems both for bandwidth aggregation as well as metadata distribution.
PLFS development continues today to expand PLFS functionality for an exa-
scale future in which POSIX is finally replaced with a more parallel-amenable
