Database Reference
In-Depth Information
0001 1010 0001 1101 1100 0100 1010 1110 0101 1100 1101 0011 0001 1010 1101 1100
Figure 5-10
A splittable compressed text ile
Improved Workload Scheduling:
Intelligent Scheduler
Open source Hadoop ships with a rudimentary first in, first out (FIFO) sched-
uler and a pluggable architecture supporting alternate scheduling options.
There are two pluggable scheduling tools available through the Apache Hadoop
project: the Fair Scheduler and the Capacity Scheduler. These schedulers are
similar in that they enable a minimum level of resources to be available for
smaller jobs to avoid starvation. These schedulers don't provide adequate
controls to ensure optimal cluster performance or offer administrators the
flexibility that they need to implement customizable workload management.
For example, Fair is pretty good at ensuring that resources are applied to
workloads, but it doesn't give you SLA-like granular controls.
As you might imagine, IBM has decades of experience and research exper-
tise in workload management. From the mainframe to distributed systems,
there are continually repeating proof points of this expertise. For example,
IBM is the only database vendor that we know of to integrate database-level
workload management (in DB2) with the underlying host operating system's
workload management capabilities such that they work in unison. Workload
management for data in a database or data in a file system is still workload
management.
Performance experts in IBM Research have studied the workload schedul-
ing problems in Hadoop and have crafted a solution called the Intelligent
Scheduler (previously known as the FLEX Scheduler). This scheduler extends
the Fair Scheduler and manipulates it by constantly altering the minimum
number of slots that are assigned to jobs. The Intelligent Scheduler includes a
variety of metrics that you can use to optimize your workloads. These metrics
can be chosen by an administrator on a cluster-wide basis, or by individual
