Database Reference
In-Depth Information
you need, you could buy 200 $5,000 machines instead. Google engineers
thought, “Surely there is a way we could put those 200 servers to work so
that the next time we need to increase the size, we just need to buy a few
more cheap machines, rather than upgrade to the $5 million server.” Their
solution was to scale out, rather than scale up.
Big Data Stack 1.0
Between 2000 and 2004, armed with a few principles, Google laid the
foundation for its Big Data strategy:
• Anything can fail, at any time, so write your software expecting
unreliable hardware. At most companies, when a database server
crashes, it is a serious event. If a network switch dies, it will probably
cause downtime. By running in an environment in which individual
components fail often, you paradoxically end up with a much more
stable system because your software is designed to handle those
failures. You can quantify your risk beyond blindly quoting statistics,
such as mean time between failures (MTBFs) or service-level
agreements (SLAs).
• Use only commodity, off-the-shelf components. This has a number of
advantages: You don't get locked into a particular vendor's feature set;
you can always find replacements; and you don't experience big price
discontinuities when you upgrade to the “bigger” version.
• The cost for twice the amount of capacity should not be considerably
more than the cost for twice the amount of hardware. This means the
software must be built to scale out, rather than up. However, this also
imposes limits on the types of operations that you can do. For instance,
if you scale out your database, it may be difficult to do a
JOIN
operation, since you'd need to join data together that lives on different
machines.
• “A foolish consistency is the hobgoblin of little minds.” If you abandon
the “C” (consistency) in ACID database operations, it becomes much
easier to parallelize operations. This has a cost, however; loss of
consistency means that programmers have to handle cases in which
reading data they just wrote might return a stale (inconsistent) copy.
This means you need smart programmers.
