Database Reference
In-Depth Information
- It identifies an approach for MapReduce frameworks to improve performance
on clusters that contain nodes with non-uniform processing capabilities.
- It provides evidence that upgrades to a cluster that do not improve all nodes
of the cluster uniformly can have a range of impacts on the turnaround time of
MapReduce applications, suggesting that data center managers should care-
fully consider upgrades. These considerations should be made based upon
both the techniques employed by the MapReduce framework to respond to
heterogeneity, and the applications the framework runs most frequently.
The paper proceeds as follows: Sect.
2
describes related work, and Sect.
3
pro-
vides relevant background on our MARLA MapReduce framework, including
the mechanisms for dividing and sub-dividing tasks at runtime. Section
4
then
describes our testbed and experiments. Section
5
includes and analyzes results
from experiments running on clusters containing two levels of processing nodes
in varying percentages. Section
6
describes results for clusters that exhibit three
levels of node capability. Sections
8
describes our plans for future work.
This work is an extension of the conference publication in the IEEE BigData
Conference, Research Track [
13
]. The significant extensions in this submission
are the following: Sect.
5.2
has been added for a detailed discussion on the perfor-
mance effect of Progressive Granularity Changes. Figure
2
shows the execution
times when we follow a two tasks per worker splitting rule. During these tests
we incrementally perform upgrades on a subset of the nodes. Figure
3
shows the
results when we follow a two tasks per worker scenario as the cluster is incremen-
tally upgraded. In particular these results are presented relative to the execution
time of an un-upgraded cluster. Figure
4
shows these same relative results when
we follow a three tasks per worker splitting rule. Additionally we consider the
impact of all of our experimentation relative to the data size involved. These
results are considered in Fig.
12
and discussed in Sect.
7
.
2 Related Work
Zaharia et al. [
5
]andXieetal.[
6
] address MapReduce job scheduling in hetero-
geneous clusters. During speculative execution, Hadoop's
straggler
mechanism
starts duplicate tasks when it discovers slow nodes, but this strategy falls short of
solving the problem in clusters with increased performance-heterogeneity. Fadika
et al. [
10
] show that Hadoop's straggler mitigation scheme also falls short in non-
dedicated clusters with third-party load.
The LATE scheduler [
5
] allows Hadoop to speculatively execute the task
that the framework expects to complete furthest into the future. LATE relies
on HDFS for data placement and therefore can not delay the binding of data to
worker processes; this restriction limits the tasks that may benefit from specu-
lative execution.
Xie et al. [
6
] profile cluster node capabilities and skew data partitioning
accordingly; slow nodes receive less work than faster nodes. This static profil-
ing suffers when “surprise” third-party loads begin or end in the middle of a
MapReduce job, thereby altering a node's ability to complete work compared to
Search WWH ::
Custom Search