Hardware Reference
In-Depth Information
TABLE 2.1: NERSC storage systems noting storage type, bandwidth capa-
bilities, and storage capacity.
Storage System
Name
Purpose
Type
Total
Bandwidth
Total
Capacity
Edison
scratch
Local
bandwidth
Lustre
140 GB/s
6.4 PB
1-3
Hopper
scratch
Local
bandwidth
Lustre
72 GB/s
2.2 PB
1-2
Global scratch
Global
bandwidth
GPFS
80 GB/s
4 PB
Global project
Global capacity
GPFS
40 GB/s
5 PB
Global projectb
Global
bandwidth
GPFS
30 GB/s
2.6 PB
Global dna
Global capacity
GPFS
12 GB/s
1.1 PB
Global common
Global
throughput
GPFS
NA
6 TB
Global homes
Global
throughput
GPFS
NA
250 TB
Archive
Long-term
capacity
HPSS
12 GB/s
240 PB
12 weeks. These file systems provide the highest bandwidth at NERSC to a
single computational system and are primarily used for output of all jobs on
the computational system. For example, details of the Edison system's local
scratch file system design are provided in Figure 2.2. The Edison scratch file
systems are Lustre file systems and achieve over 140-GB/s bandwidth in aggre-
gate. To maximize metadata performance in the scratch file system, Edison
deploys three separate scratch file systems. Two of the scratch file systems
have a bandwidth of 35 GB/s and a storage capacity of 1.6 PB with users
evenly divided on each, therefore a user either uses
/scratch1
or
/scratch2
.
The third scratch file system,
/scratch3
, has a 70-GB/s bandwidth and 3.2
PB of storage capacity. The
/scratch3
file system is a schedulable resource
available to all users of the Edison system.
In order to externalize each local scratch file system from the compu-
tational system so that storage may remain operational to login nodes and
other assets at the facility during computational system downtime, the LNET
routers' purpose is to route storage trac between the Edison Aries intercon-
nect [3] and the file system fourteen data rate (FDR) InfiniBand (IB) network.
The IB connections are provisioned to drive each file system at its maximum
35-GB/s bandwidth, and connect each LNET router to top-of-rack switches
in each file system cabinet. The file system hardware makes use of embedded
storage such that the object storage targets (OSTs) are run on embedded
servers inside the storage enclosures.
