Database Reference
In-Depth Information
The
queuePlacementPolicy
can be omitted entirely, in which case the default beha-
vior is as if it had been specified with the following:
<queuePlacementPolicy>
<rule
name=
"specified"
/>
<rule
name=
"user"
/>
</queuePlacementPolicy>
In other words, unless the queue is explicitly specified, the user's name is used for the
queue, creating it if necessary.
Another simple queue placement policy is one where all applications are placed in the
same (default) queue. This allows resources to be shared fairly between applications,
rather than users. The definition is equivalent to this:
<queuePlacementPolicy>
<rule
name=
"default"
/>
</queuePlacementPolicy>
It's also possible to set this policy without using an allocation file, by setting
yarn.scheduler.fair.user-as-default-queue
to
false
so that applica-
tions will be placed in the default queue rather than a per-user queue. In addition,
yarn.scheduler.fair.allow-undeclared-pools
should be set to
false
so
that users can't create queues on the fly.
Preemption
When a job is submitted to an empty queue on a busy cluster, the job cannot start until re-
sources free up from jobs that are already running on the cluster. To make the time taken
for a job to start more predictable, the Fair Scheduler supports
preemption
.
Preemption allows the scheduler to kill containers for queues that are running with more
than their fair share of resources so that the resources can be allocated to a queue that is
under its fair share. Note that preemption reduces overall cluster efficiency, since the ter-
minated containers need to be reexecuted.
Preemption is enabled globally by setting
yarn.scheduler.fair.preemption
to
true
. There are two relevant preemption timeout settings: one for minimum share and
one for fair share, both specified in seconds. By default, the timeouts are not set, so you
need to set at least one to allow containers to be preempted.
If a queue waits for as long as its
minimum share preemption timeout
without receiving its
minimum guaranteed share, then the scheduler may preempt other containers. The default
timeout is set for all queues via the
defaultMinSharePreemptionTimeout
top-