Database Reference
In-Depth Information
Scheduling in YARN
In an ideal world, the requests that a YARN application makes would be granted immedi-
ately. In the real world, however, resources are limited, and on a busy cluster, an applica-
tion will often need to wait to have some of its requests fulfilled. It is the job of the YARN
scheduler to allocate resources to applications according to some defined policy. Schedul-
ing in general is a difficult problem and there is no one “best” policy, which is why YARN
provides a choice of schedulers and configurable policies. We look at these next.
Scheduler Options
Three schedulers are available in YARN: the FIFO, Capacity, and Fair Schedulers. The
FIFO Scheduler places applications in a queue and runs them in the order of submission
(first in, first out). Requests for the first application in the queue are allocated first; once its
requests have been satisfied, the next application in the queue is served, and so on.
The FIFO Scheduler has the merit of being simple to understand and not needing any con-
figuration, but it's not suitable for shared clusters. Large applications will use all the re-
sources in a cluster, so each application has to wait its turn. On a shared cluster it is better
to use the Capacity Scheduler or the Fair Scheduler. Both of these allow long-running jobs
to complete in a timely manner, while still allowing users who are running concurrent
smaller ad hoc queries to get results back in a reasonable time.
The difference between schedulers is illustrated in
Figure 4-3
, which shows that under the
FIFO Scheduler (i) the small job is blocked until the large job completes.
With the Capacity Scheduler (ii in
Figure 4-3
)
, a separate dedicated queue allows the small
job to start as soon as it is submitted, although this is at the cost of overall cluster utiliza-
tion since the queue capacity is reserved for jobs in that queue. This means that the large
job finishes later than when using the FIFO Scheduler.
With the Fair Scheduler (iii in
Figure 4-3
)
, there is no need to reserve a set amount of capa-
city, since it will dynamically balance resources between all running jobs. Just after the first
(large) job starts, it is the only job running, so it gets all the resources in the cluster. When
the second (small) job starts, it is allocated half of the cluster resources so that each job is
using its fair share of resources.
Note that there is a lag between the time the second job starts and when it receives its fair
share, since it has to wait for resources to free up as containers used by the first job com-
plete. After the small job completes and no longer requires resources, the large job goes
back to using the full cluster capacity again. The overall effect is both high cluster utiliza-
tion and timely small job completion.
