Database Reference
In-Depth Information
The queuePlacementPolicy can be omitted entirely, in which case the default beha-
vior is as if it had been specified with the following:
<queuePlacementPolicy>
<rule name= "specified" />
<rule name= "user" />
</queuePlacementPolicy>
In other words, unless the queue is explicitly specified, the user's name is used for the
queue, creating it if necessary.
Another simple queue placement policy is one where all applications are placed in the
same (default) queue. This allows resources to be shared fairly between applications,
rather than users. The definition is equivalent to this:
<queuePlacementPolicy>
<rule name= "default" />
</queuePlacementPolicy>
It's also possible to set this policy without using an allocation file, by setting
yarn.scheduler.fair.user-as-default-queue to false so that applica-
tions will be placed in the default queue rather than a per-user queue. In addition,
yarn.scheduler.fair.allow-undeclared-pools should be set to false so
that users can't create queues on the fly.
Preemption
When a job is submitted to an empty queue on a busy cluster, the job cannot start until re-
sources free up from jobs that are already running on the cluster. To make the time taken
for a job to start more predictable, the Fair Scheduler supports preemption .
Preemption allows the scheduler to kill containers for queues that are running with more
than their fair share of resources so that the resources can be allocated to a queue that is
under its fair share. Note that preemption reduces overall cluster efficiency, since the ter-
minated containers need to be reexecuted.
Preemption is enabled globally by setting yarn.scheduler.fair.preemption to
true . There are two relevant preemption timeout settings: one for minimum share and
one for fair share, both specified in seconds. By default, the timeouts are not set, so you
need to set at least one to allow containers to be preempted.
If a queue waits for as long as its minimum share preemption timeout without receiving its
minimum guaranteed share, then the scheduler may preempt other containers. The default
timeout is set for all queues via the defaultMinSharePreemptionTimeout top-
Search WWH ::




Custom Search