Hardware Reference
In-Depth Information
Chapter 33
Storage Networks and Interconnects
Parks Fields and Benjamin McClelland
Los Alamos National Laboratory
33.1
Current State of Technology ::::::::::::::::::::::::::::::::::::: 363
33.2
Future Directions :::::::::::::::::::::::::::::::::::::::::::::::: 365
33.3
Challenges and Solutions :::::::::::::::::::::::::::::::::::::::: 366
33.4
Conclusion :::::::::::::::::::::::::::::::::::::::::::::::::::::::: 367
Bibliography :::::::::::::::::::::::::::::::::::::::::::::::::::::: 367
33.1 Current State of Technology
At present, HPC facilities primarily use Ethernet and IB interconnect tech-
nologies, although Cray and IBM have deployed some larger systems with
a proprietary interconnect. In the June 2013 Top500 list [8], 40.2% of the
systems use InfiniBand (DDR/QDR/FDR) as the interconnect, 43% use a
combination of 1 GigE and 10 GigE, 4.2% use the Cray interconnect (Seast-
ar/Gemini/Aries), 2% are proprietary, and 10% are custom interconnects.
The remaining 4.2% of the proprietary interconnects that are not using Cray
on the Top500 are from IBM and other countries like China and Japan who
are developing their own interconnects. The custom category includes inter-
connects like the tofu network found on the Fujitsu machine. InfiniBand has
proven itself over the years for small- and medium-size clusters. But due to
scalability concerns, the DOE National Laboratories have avoided using IB
for their largest capability machines.
The InfiniBand networks are usually created with a Fat Tree topology,
but there are some exceptions. For example, REDSKY at Sandia National
Lab was implemented with a 3D torus [7]. Sandia contributed much work and
software to the routing algorithm to make sure that the network was deadlock
free and avoided credit loops. Silicon Graphics International Corp. has used a
HyperCube topology in some of the larger IB machines. The IB network has
choices for routing algorithms, such as Fat Tree, Up-Down, Min-Hop, LASH,
and DOR [9].
The IB market is primarily composed of two vendors, Mellanox and
QLogic. Mellanox supplied chips to other companies including Voltaire.
363
