YARN - Hadoop: The Definitive Guide - page 110

are divided evenly. Weights are not quite the same as percentages, even though the ex-
ample uses numbers that add up to 100 for the sake of simplicity. We could have specified
weights of 2 and 3 for the prod and dev queues to achieve the same queue weighting.
NOTE
When setting weights, remember to consider the default queue and dynamically created queues (such as
queues named after users). These are not specified in the allocation file, but still have weight 1.
Queues can have different scheduling policies. The default policy for queues can be set in
the top-level defaultQueueSchedulingPolicy element; if it is omitted, fair
scheduling is used. Despite its name, the Fair Scheduler also supports a FIFO ( fifo )
policy on queues, as well as Dominant Resource Fairness ( drf ), described later in the
chapter.
The policy for a particular queue can be overridden using the schedulingPolicy ele-
ment for that queue. In this case, the prod queue uses FIFO scheduling since we want
each production job to run serially and complete in the shortest possible amount of time.
Note that fair sharing is still used to divide resources between the prod and dev queues,
as well as between (and within) the eng and science queues.
Although not shown in this allocation file, queues can be configured with minimum and
maximum resources, and a maximum number of running applications. (See the reference
page for details.) The minimum resources setting is not a hard limit, but rather is used by
the scheduler to prioritize resource allocations. If two queues are below their fair share,
then the one that is furthest below its minimum is allocated resources first. The minimum
resource setting is also used for preemption, discussed momentarily.
Queue placement
The Fair Scheduler uses a rules-based system to determine which queue an application is
placed in. In Example 4-2 , the queuePlacementPolicy element contains a list of
rules, each of which is tried in turn until a match occurs. The first rule, specified ,
places an application in the queue it specified; if none is specified, or if the specified
queue doesn't exist, then the rule doesn't match and the next rule is tried. The
primaryGroup rule tries to place an application in a queue with the name of the user's
primary Unix group; if there is no such queue, rather than creating it, the next rule is tried.
The default rule is a catch-all and always places the application in the dev.eng
queue.