Hardware Reference
In-Depth Information
FIGURE 6.1: LANL Parallel Scalable Backbone, as invented a decade ago.
The compute nodes route data via their cluster interconnect fabric to an
appropriate I/O node which then routes via a Lane Ethernet switch to a set
of parallel GPFSs. IP round-robin routing with failover is used in the I/O node
to route around bad I/O nodes. Dead gateway detection is used on the I/O
nodes to find and kill bad I/O nodes or switches, and Equal Cost Multi-Path
(ECMP) routing is used to balance the work across I/O nodes and Ethernet
paths. [Image courtesy of Gary Grider (LANL).]
storage server, which enabled site-wide sharing|a great asset that was reason-
ably low cost. The multiple metadata servers for one file system was also very
useful to isolate important users on their own metadata servers. The declus-
tered RAID [7] and scalable rebuild was one of the most important features
that kept LANL's systems available for years. Declustered RAID reads from
many disks and writes the rebuilt parts to many other disks, which means the
rebuild process can go as fast as many disks (therefore scaling by the number
of disks one has). Also, it maintained the ability to run a reconstruction on
portions of a huge le system while keeping the le system available. Panasas's
management interface was ahead of its time as well. Many of its features are
just now showing up in other parallel le systems. More recently, for LANL's
largest capability machine, they have a Lustre parallel file system which is
working well. The hardware includes NetApp/LSI RAID disk systems with
the availability of 10 PB and 160 GB/s.
