Hardware Reference
In-Depth Information
storage subsystem integrates Scalable Storage Units (SSU) that each has two
industry-standard x86 Embedded Server Modules (ESMs) based on the Intel
Sandybridge
TM
processors. The ESMs connect directly through a common mid-
plane to all drives in the SSU and share a redundant high-speed interconnect
across the midplane for failover services. The ESMs run an industry-standard
Linux distribution and each module has its own dedicated CPU, memory,
network, and storage connectivity.
Each SSU contains eighty-two, 2-TB, 7200-RPM NL-SAS disk drives
housed in two trays. Eighty data drives are configured in eight RAID 6 (8+2)
arrays resulting in 64 usable data drives. The other two drives in the SSU are
global hot spares. Each SSU has two 100-GB SSDs for OSS metadata and two
OSS servers. Together, the metadata and object storage servers present file
systems to clients.
The Sonexion CS-1600 Management Server (MGS) stores configuration
information for all Lustre file systems in Blue Waters. Each Lustre server
and each Lustre client rely on the MGS to provide information. Metadata
Servers (MDS) make metadata available to Lustre clients from the Metadata
Target (MDT). The MDT stores file system metadata (filenames, directories,
permissions, and file layouts) on disk and manages the namespace. The MDS
provides network request handling for the file system. The OSSs enable file
I/O service and network request handling for one or more local OSTs. The
OST stores data (files or chunks of files) on a single LUN (disk drive or an
array of disk drives). Cray XE6/XK7 Lustre clients, called LNETs, interface
user applications to the Lustre file system. The client presents a POSIX file
system interface to user-level applications running on compute/analysis nodes
and is responsible for routing I/O requests, data, and metadata through the
Cray XE6/XK7 system to the Lustre servers in the CS-1600 storage cluster.
Cray's XE6/XK7 LNETs implement the Lustre networking protocol between
the Lustre client and the Lustre server. The LNET layer allows clients and
servers to be executed on the same system (i.e., direct attached Lustre) or
for compute clients to communicate with external Lustre servers or Lustre
appliances.
3.2.1 On-line Storage Performance
The on-line storage subsystem provides over 25 usable (36 raw) petabytes
of storage to the science and engineering teams. The overall bandwidth ex-
ceeds 1.18 TB/s as measured by a range of tests [1]. Similarly, the measured
metadata performance is over 25,000 mean creates per second of a single file
from each node; 30,000 concurrent deletes per second and 40,000
stat()
calls
per second as aggregate rates using up to all XE6 client nodes in either a
single directory or in separate directories.
To demonstrate Blue Waters' I/O capability, Table 3.3 lists observed re-
sults from executions of the IOR benchmark doing a \write" operation on each
of its three file systems. Each value in the table corresponds to the mean of
