Hardware Reference
In-Depth Information
2-4 GB SDRAM
Set of
8 boards
Card
cage
Card
cage
Opteron
Card
cage
100-Mbps Ethernet
Seastar
Cabinet
Figure 8-40.
Packaging of the Red Storm components.
SDRAM. Each CPU has access only to its own SDRAM. There is no shared
memory. The theoretical computing power of the system is 41 teraflops/sec.
The interconnection between the Opteron CPUs is done by the custom Seastar
routers, one router per Opteron CPU. They are connected in a 3D torus of size
27
24 with one Seastar at each mesh point. Each Seastar has seven bidirec-
tional 24-Gbps links, going north, east, south, west, up, down, and to the Opteron.
The transit time between adjacent mesh points is 2 microsec. Across the entire set
of compute nodes it is only 5 microsec. A second network using 100-Mbps Ether-
net is used for service and maintenance.
In addition to the 108 compute cabinets, the system also contains 16 cabinets
for I/O and service processors. Each cabinet holds 32 Opterons. These 512 CPUs
are split: 256 for I/O and 256 for service. The rest of the space is for disks, which
are organized as RAID 3 and RAID 5, each with a parity drive and a hot spare.
The total disk space is 240 TB. The combined disk bandwidth is 50 GB/sec.
The system is partitioned into classified and unclassified sections, with switch-
es between the parts so they can be mechanically coupled or decoupled. A total of
2688 are always in the classified section and another 2688 Opterons are always in
the unclassified section. The remaining 4992 Opterons are switchable into either
section, as depicted in Fig. 8-41. The 2688 classified Opterons each have 4 GB of
RAM; all the rest have 2 GB each. Apparently classified work is memory inten-
sive. The I/O and service processors are split between the two parts.
×
16
×
