Database Reference
In-Depth Information
input. The Reduce task executes code written by the user and writes its output to a file that
is part of the surrounding distributed file system.
2.2.6
Coping With Node Failures
The worst thing that can happen is that the compute node at which the Master is executing
fails. In this case, the entire MapReduce job must be restarted. But only this one node
can bring the entire process down; other failures will be managed by the Master, and the
MapReduce job will complete eventually.
Suppose the compute node at which a Map worker resides fails. This failure will be de-
tected by the Master, because it periodically pings the Worker processes. All the Map tasks
that were assigned to this Worker will have to be redone, even if they had completed. The
reason for redoing completed Map tasks is that their output destined for the Reduce tasks
resides at that compute node, and is now unavailable to the Reduce tasks. The Master sets
the status of each of these Map tasks to idle and will schedule them on a Worker when one
becomes available. The Master must also inform each Reduce task that the location of its
input from that Map task has changed.
Dealing with a failure at the node of a Reduce worker is simpler. The Master simply sets
the status of its currently executing Reduce tasks to idle. These will be rescheduled on an-
other reduce worker later.
2.2.7
Exercises for Section 2.2
EXERCISE
2.2.1 Suppose we execute the word-count MapReduce program described in this
section on a large repository such as a copy of the Web. We shall use 100 Map tasks and
some number of Reduce tasks.
(a) Suppose we do not use a combiner at the Map tasks. Do you expect there to be sig-
nificant skew in the times taken by the various reducers to process their value list?
Why or why not?
(b) If we combine the reducers into a small number of Reduce tasks, say 10 tasks, at
random, do you expect the skew to be significant? What if we instead combine the
reducers into 10,000 Reduce tasks?
! (c)
Suppose we do use a combiner at the 100 Map tasks. Do you expect skew to be
significant? Why or why not?
