Hardware Reference
In-Depth Information
Servers In A Google WSC
The server in
Figure 6.21
has two sockets, each containing a dual-core AMD Opteron processor
running at 2.2 GHz. The photo shows eight DIMMS, and these servers are typically deployed
with 8 GB of DDR2 DRAM. A novel feature is that the memory bus is downclocked to 533
MHz from the standard 666 MHz since the slower bus has litle impact on performance but a
signiicant impact on power.
The baseline design has a single network interface card (NIC) for a 1 Gbit/sec Ethernet link.
Although the photo in
Figure 6.21
shows two SATA disk drives, the baseline server has just
one. The peak power of the baseline is about 160 wats, and idle power is 85 wats.
This baseline node is supplemented to offer a storage (or “diskfull”) node. First, a second
tray containing 10 SATA disks is connected to the server. To get one more disk, a second disk
is placed into the empty spot on the motherboard, giving the storage node 12 SATA disks. Fin-
ally, since a storage node could saturate a single 1 Gbit/sec Ethernet link, a second Ethernet
NIC was added. Peak power for a storage node is about 300 wats, and it idles at 198 wats.
Note that the storage node takes up two slots in the rack, which is one reason why Google
deployed 40,000 instead of 52,200 servers in the 45 containers. In this facility, the ratio was
about two compute nodes for every storage node, but that ratio varied widely across Google's
WSCs. Hence, Google A had about 190,000 disks in 2007, or an average of almost 5 disks per
server.
Networking In A Google WSC
The 40,000 servers are divided into three arrays of more than 10,000 servers each. (Arrays are
called
clusters
in Google terminology.) The 48-port rack switch uses 40 ports to connect to serv-
ers, leaving 8 for uplinks to the array switches.
Array switches are configured to support up to 480 1 Gbit/sec Ethernet links and a few 10
Gbit/sec ports. The 1 Gigabit ports are used to connect to the rack switches, as each rack switch
has a single link to each of the array switches. The 10 Gbit/sec ports connect to each of two
datacenter routers, which aggregate all array routers and provide connectivity to the outside
world. The WSC uses two datacenter routers for dependability, so a single datacenter router
failure does not take out the whole WSC.
The number of uplink ports used per rack switch varies from a minimum of 2 to a maximum
of 8. In the dual-port case, rack switches operate at an oversubscription rate of 20:1. That is,
there is 20 times the network bandwidth inside the switch as there was exiting the switch. Ap-
plications with significant traffic demands beyond a rack tended to suffer from poor network
performance. Hence, the 8-port uplink design, which provided a lower oversubscription rate
of just 5:1, was used for arrays with more demanding traffic requirements.
Monitoring And Repair In A Google WSC
For a single operator to be responsible for more than 1000 servers, you need an extensive mon-
itoring infrastructure and some automation to help with routine events.
Google deploys monitoring software to track the health of all servers and networking gear.
Diagnostics are running all the time. When a system fails, many of the possible problems have
simple automated solutions. In this case, the next step is to reboot the system and then to try
to reinstall software components. Thus, the procedure handles the majority of the failures.
Machines that fail these first steps are added to a queue of machines to be repaired. The dia-
gnosis of the problem is placed into the queue along with the ID of the failed machine.
Search WWH ::
Custom Search