Database Reference
In-Depth Information
Cloud Data Warehousing
Most companies are accustomed to storing their data on-premise or in
leased datacenters on hardware that they own or rent. Fault tolerance is
usually handled by adding redundancy within a machine, such as extra
power supplies, RAID disk controllers, and ECC memory. All these things
add to the cost of the machine but don't actually distance you from the
consequences of a hardware failure. If a disk goes bad, someone has to go
to the datacenter, find the rack with the bad disk, and swap it out for a new
one.
Cloud data warehousing offers the promise of relieving you of the
responsibility of caring about whether RAID-5 is good enough, whether
your tape backups are running frequently enough, or whether a natural
disaster might take you offline completely. Cloud data warehouses, whether
Google's or a competitor's, offer fault-tolerance, geographic distribution,
and automated backups.
Ever since Google made the decision to go with exclusively scale-out
architectures, it has focused on making its software accustomed to handling
frequent hardware failures. There are stories about Google teams that run
mission-critical components, who don't even bother to free memory—the
amount of bugs and performance problems associated with memory
management is too high. Instead, they just let the process run out of
memory and crash, at which time it will get automatically restarted. Because
the software has been designed to not only handle but also expect that type
of failure, a large class of errors is virtually eliminated.
For the user of Google's cloud, this means that the underlying infrastructure
pieces are extraordinarily failure-resistant and fault-tolerant. Your data is
replicated to several disks within a datacenter and then replicated again
to multiple datacenters. Failure of a disk, a switch, a load balancer, or a
rack won't be noticeable to anyone except a datacenter technician. The only
kind of hardware failure that would escalate to the BigQuery operations
engineers would be if someone hit the big red off button in a datacenter or
if somebody took out a fiber backbone with a backhoe. This type of failure
still wouldn't take BigQuery down, however, since BigQuery runs in multiple
geographically distributed datacenters and will fail over automatically.
Of course, this is where we have to remind you that all software is fallible.
Just because your data is replicated nine ways doesn't mean that it is
