Hardware Reference
In-Depth Information
FIGURE 2.2: A diagram showing three different scratch file systems and con-
nections via IB to the Edison compute nodes. [Image courtesy of Jason Hick
(NERSC).]
Users are generally pleased with the performance and stability of the Cen-
ter's local scratch le systems, with two typical points of improvement: meta-
data operation responsiveness and increasing bandwidth.
2.2.2 Storage Network
Starting more than a decade ago, NERSC established one of the largest
HPC fiber channels (FCs)-based storage area networks (SANs). Although fiber
channel technology was highly priced, it was the only truly lossless and full-
bandwidth transport medium offered for storage systems. The FC SAN pro-
vided near full bandwidth access to storage in the NGF file systems from every
computational system at the facility. However, the FC capacity increase from
8 Gigabits (Gb) to 16 Gb took a long time to be generally available. In the
meantime, 40 Gb IB became stable and more cost effective. Though IB tech-
nology still lacks significant features that would make it an obvious choice for
next-generation storage area networks, many HPC facilities, NERSC included,
are building next-generation storage networks using IB technology.
Many facilities selected Ethernet as their storage protocol for providing
bandwidth, but spent a significant effort tuning it to approach streaming
performance over a lossy network technology. Storage systems require lossless
network technology for achieving their full bandwidth capability. The NGF
file systems have a secondary, failover, 10 Gigabit Ethernet (GigE) network
interface. Having the capability to use distinct failover networks is a significant
feature of the General Parallel File System (GPFS), discussed in Chapter 9.
For the centerwide file systems, many of the computational systems at
the facility have I/O nodes that help route trac between the computational
system interconnect (a mix of proprietary and IB connections) and the SAN,
which is a mix between FC and IB. The centerwide file system is redundantly
 
Search WWH ::




Custom Search