Hardware Reference
In-Depth Information
operations executed by other clients (e.g., create within a directory created by
the failing client) are also lost. This can be eliminated by enabling Commit-on-
Share (COS), which flushes relevant uncommitted state at the server before
a change of DLM lock ownership completes.
8.3 Deployment and Usage
Since Lustre is an open-source file system, it can run on a huge variety of
different platforms. There is no vendor lock-in and storage purchases can be
made to best suit the buyer's requirements and budget. Lustre therefore pro-
vides the foundation for a healthy ecosystem of key storage vendors including
among others, DataDirect Networks, EMC, Hitachi Data Systems, NetApp
and Xyratex. Such variety makes it hard to collect precise data about the
number of deployments in production, but a conservative estimate would be
in the thousands worldwide.
Many sites use Lustre as global storage, shared site-wide across multiple
computing resources from supercomputers through visualization and analysis
clusters down to individual workstations. Lustre routers provide the bridge
between the HPC fabrics used internally within supercomputing clusters and
the storage network|often Inniband or high performance Ethernet|used
site-wide. This eliminates the \islands of data" associated with previous HPC
file systems and enables workflows to span all the computing resources at a
site without having to copy data unnecessarily [2].
Lustre's origins from research funded by US government labs has meant
that widespread use was first established in the research and academic sectors.
They used Lustre primarily for checkpointing and to store simulation and
modeling datasets. However, as the product has matured, a growing number
of organizations in the private sector have adopted Lustre, particularly in
genomics and in the oil and gas industry.
The oil and gas industry relies heavily on seismic data for the exploration
and appraisal of new oil and gas reserves. Single datasets can stretch to hun-
dreds of terabytes in size and require dedicated high performance computing
resources for processing and interpretation. Lustre is attractive in this field
since it is significantly speeding up the processing times, it presents a single
global file system namespace, and additional storage can be added without
needing to take the system out of service.
Large volumes of data are also a factor in geonomics. New improvements in
laboratory technologies and processes continue to increase the demand at an
astonishing rate. For example, the Sanger Institute reports that they produce
ten thousand times the amount of sequencing data compared to five years
ago, therefore the ability provided by Lustre to expand seamlessly, both in
performance and capacity, to meet evolving capacity demands is crucial.
