Hardware Reference
In-Depth Information
FIGURE 33.1: The IBTA's InniBand roadmap from 2005 to 2014. [Image
courtesy of InfiniBand Trade Association.]
Recently, Intel bought the QLogic IB team and their intellectual property
(IP) for IB and is currently selling the QDR hardware [2].
IB has a very good install base and has a significant open source soft-
ware community. IB is following various portions of the IB Trade Association
(IBTA) roadmap as seen in Figure 33.1. The IB vendors have rarely imple-
mented IB 12 preferring the 4 path, as it is easier to implement and more
profitable. The IB community has slowed its original aggressive bandwidth
pace and adjusted to market demand.
There are many cluster vendors integrating Ethernet as the primary inter-
connect. Because Gigabit Ethernet is so cheap, many companies use it where
bandwidth and latency between nodes is not as important. Many companies
sell clusters with 10 GigE as the primary interconnect also. Chip makers such
as Broadcom and Fulcrum created low-latency Ethernet chips that only add
a few nanoseconds per hop. Gnodal is an Ethernet company that has cho-
sen to design its own ASIC, which handles congestion very well. There are
Remote Direct Memory Access (RDMA) implementations that function over
Ethernet including RDMA over Converged Enhanced Ethernet (RoCEE) and
Internet Wide Area RDMA Protocol (iWARP). However, due to bandwidth
and latency issues when there is congestion, the DOE National Labs have not
adopted this technology as a cluster interconnect.
Originally developed by Cray and then purchased by Intel in 2012, the
Aries interconnect was the fastest interconnect as of 2013. The Aries network
is PCIe-based but only available on a Cray built node. In 2012 Cray sold
the interconnect IP and hardware team to Intel with the ability to continue
