Database Reference
In-Depth Information
Queue configuration
The Fair Scheduler is configured using an allocation file named
fair-scheduler.xml
that is
loaded from the classpath. (The name can be changed by setting the property
yarn.scheduler.fair.allocation.file
.) In the absence of an allocation file,
the Fair Scheduler operates as described earlier: each application is placed in a queue
named after the user and queues are created dynamically when users submit their first ap-
plications.
Per-queue configuration is specified in the allocation file. This allows configuration of
hierarchical queues like those supported by the Capacity Scheduler. For example, we can
define
prod
and
dev
queues like we did for the Capacity Scheduler using the allocation
file in
Example 4-2
.
Example 4-2. An allocation file for the Fair Scheduler
<?xml version="1.0"?>
<allocations>
<defaultQueueSchedulingPolicy>
fair
</defaultQueueSchedulingPolicy>
<queue
name=
"prod"
>
<weight>
40
</weight>
<schedulingPolicy>
fifo
</schedulingPolicy>
</queue>
<queue
name=
"dev"
>
<weight>
60
</weight>
<queue
name=
"eng"
/>
<queue
name=
"science"
/>
</queue>
<queuePlacementPolicy>
<rule
name=
"specified"
create=
"false"
/>
<rule
name=
"primaryGroup"
create=
"false"
/>
<rule
name=
"default"
queue=
"dev.eng"
/>
</queuePlacementPolicy>
</allocations>
The queue hierarchy is defined using nested
queue
elements. All queues are children of
the
root
queue, even if not actually nested in a
root queue
element. Here we subdi-
vide the
dev
queue into a queue called
eng
and another called
science
.
Queues can have weights, which are used in the fair share calculation. In this example, the
cluster allocation is considered fair when it is divided into a 40:60 proportion between
prod
and
dev
. The
eng
and
science
queues do not have weights specified, so they
