Hardware Reference
In-Depth Information
FIGURE 4.2: The ALCF2 HPC system.
of Intrepid. The I/O nodes on Mira are again the same processor. However,
the processes are in separate dedicated drawers and are air cooled rather than
liquid cooled. Further, rather than having the I/O on chip, there is a PCIe v2.0
8 slots available. Each slot was equipped with a Mellanox ConnectX 2 QDR
(40 Gb/s) Infiniband card. The storage fabric consists of four large Mellanox
IS5600 648-port switch chassis. The four core switches each have 324 ports of
inter-switch links, with 108 ports going to each of the other three switches.
The remaining edge ports connect to BG/Q I/O nodes, storage arrays, Tukey
compute nodes, and any other user nodes.
The storage system consists of 16 DDN Storage Fusion Architecture (SFA)
12Ke storage arrays. Like Intrepid, these are dual controllers with 10 drawers
per array, and each drawer is able to hold 60 drives. However, the controllers
are server based, rather than ASIC based. There are no external file servers.
The file servers run as virtual machines (VMs) embedded inside the storage
controller. Each controller has four VMs. Each VM has two IB ports, bonded
together as a single logical port. Capacity-wise, 56 slots in each drawer are
filled with 3 TB SATA drives, for a raw aggregate capacity of 28.8 PB. After
overheads and a small amount of space reserved for testing, the usable storage
capacity is 20 PB. Each array also contains 32 SSD units in sixteen RAID 0
LUNs. Connectivity between the controllers and drawers is via 6-Gbps SAS.
