Database Reference
In-Depth Information
users on a job-specific basis. You can optionally weight these metrics to bal-
ance competing priorities, minimize the sum of all the individual job metrics,
or maximize the sum of all of them.
The following are examples of the Intelligent Scheduler settings that you
can use to tune your workloads:
•
averageresponsetime
The scheduler allocates maximum
resources to small jobs, ensuring that these jobs are completed quickly.
•
maximumstretch
The scheduler allocates resources to jobs in
proportion to the amount of resources that they need. Big jobs have
higher priority.
•
userpriority
The scheduler allocates maximum resources to jobs
for a particular user.
Adaptive MapReduce
In their efforts to further optimize Hadoop performance, IBM Research has
developed a concept called Adaptive MapReduce, which extends Hadoop by
making individual mappers self-aware and aware of other mappers. This
approach enables individual map tasks to adapt to their environment and
make efficient decisions. (If you consider what connection pooling and con-
nection concentration did for database scalability and apply that to Adaptive
MapReduce for Hadoop, you'll have a good analogy to work from if you are
new to this space.)
In normal MapReduce, when a job is about to begin, Hadoop divides the
data into many pieces, called
splits
. Each split is assigned a single mapper. To
ensure a balanced workload, these mappers are deployed in waves, and new
mappers start after old mappers finish processing their splits. In this model, a
small split size means more mappers, which helps ensure balanced workloads
and minimizes failure costs. However, smaller splits also result in increased
cluster overhead due to the higher volumes of startup costs for each map
task. For workloads with high startup costs for map tasks, larger split sizes
tend to be more efficient. An adaptive approach to running map tasks gives
BigInsights the best of both worlds.
One component of Adaptive MapReduce is the concept of an
adaptive mapper
.
Adaptive mappers extend the capabilities of conventional Hadoop mappers
