Hardware Reference
In-Depth Information
the surveys, these storage systems were important production machines used
daily on important problems. Portions of this survey appeared in a previous
paper by Curry et al. [13].
34.2.1 Red Sky
Red Sky is the premier capacity compute platform at Sandia National
Laboratories, New Mexico. When combined with Red Mesa, a machine of the
same architecture hosted at Sandia for the National Renewable Energy Labo-
ratory (NREL), it ranked 16
th
on the June 2011 TOP500 list, having achieved
433.5 TFLOPS [3]. One of the machine's notable features is its interconnect
topology, as it is the first Infiniband machine organized as a 3D torus [19]. It
is composed of Sun X6275 blades, each containing two Intel X5570 processors
and 12 GB of RAM. The storage nodes are nearly identical to the compute
nodes, save connections to storage, and are in the same racks. Red Sky hosts
3 PB of raw storage organized as Linux software RAID 6 arrays. These provide
the storage for several Lustre file systems.
Red Sky is designed to be split between two modes of operation simulta-
neously: classified and unclassified. The unclassified partition of the machine,
was configured to be about 83% of the nodes in Red Sky, or about 54%
of the nodes in Red Sky and Red Mesa combined [29]. This portion of the
machine (called Red Sky UC from here forward) includes a 750-TB Lustre
file system for scratch, and two other 18-TB Lustre file systems for home
and project directories. Power consumption is projected for this portion of
the machine, as it is intended to be a reasonable partition that can be used
independently.
Three racks (one compute, two storage) were measured and extrapolated
to the size of Red Sky UC, which contains 40 racks of compute and six racks
of storage. Several times throughout a 14-hour period, the power use was
read directly from the compute rack's power distribution unit, and the draw
of the disk cabinets was measured directly from the circuit panel board. The
compute rack consumed between 23.2 and 27.2 kW, while the two storage racks
together consumed a steady 16 kW throughout the test period. Figure 34.1
shows power use as extrapolated to the full size of Red Sky UC. It can be
predicted that 928{1088 kW are consumed by compute and I/O nodes, while
48 kW are consumed by disks. This implies that 4.2{4.8% of the machine's
power is used for disks.
34.2.2 Cielo
Los Alamos National Laboratory and Sandia National Laboratories,
through the Alliance for Computing at Extreme Scale (ACES), procured,
operate, and use Cielo, the primary capability computing platform for both
labs [5]. Cielo is hosted at Los Alamos National Laboratory. Cielo was mea-
sured as the sixth most powerful supercomputer on the June 2011 TOP500
