Database Reference
In-Depth Information
Figure 2.1
Compute nodes are organized into racks, and racks are interconnected by a switch
It is a fact of life that components fail, and the more components, such as compute nodes
and interconnection networks, a system has, the more frequently something in the system
modes are the loss of a single node (e.g., the disk at that node crashes) and the loss of an
entire rack (e.g., the network connecting its nodes to each other and to the outside world
fails).
Some important calculations take minutes or even hours on thousands of compute nodes.
If we had to abort and restart the computation every time one component failed, then the
computation might never complete successfully. The solution to this problem takes two
forms:
(1) Files must be stored redundantly. If we did not duplicate the file at several compute
nodes, then if one node failed, all its files would be unavailable until the node is re-
placed. If we did not back up the files at all, and the disk crashes, the files would be
lost forever. We discuss file management in
Section 2.1.2
.
(2) Computations must be divided into tasks, such that if any one task fails to execute to
completion, it can be restarted without affecting other tasks. This strategy is followed
by the MapReduce programming system that we introduce in
Section 2.2
.
2.1.2
Large-Scale File-System Organization
To exploit cluster computing, files must look and behave somewhat differently from the
conventional file systems found on single computers. This new file system, often called a
distributed file system
or
DFS
(although this term has had other meanings in the past), is
typically used as follows.
• Files can be enormous, possibly a terabyte in size. If you have only small files,
there is no point using a DFS for them.
• Files are rarely updated. Rather, they are read as data for some calculation, and
possibly additional data is appended to files from time to time. For example, an
