Support Questions
Find answers, ask questions, and share your expertise

Preemption on Fair queue

Super Guru

Preemption is based on lowest priority based on last schedule app in FIFO queue. How does this work on queue setup with fair policy? It is as simple as last app which was executed within queue will be preempted

2 REPLIES 2

Guru

If a pool's minimum share is not met for some period of time, the scheduler optionally supports preemption of jobs in other pools. The pool will be allowed to kill tasks from other pools to make room to run. Preemption can be used to guarantee that "production" jobs are not starved while also allowing the Hadoop cluster to also be used for experimental and research jobs. In addition, a pool can also be allowed to preempt tasks if it is below half of its fair share for a configurable timeout (generally set larger than the minimum share preemption timeout). When choosing tasks to kill, the fair scheduler picks the most-recently-launched tasks from over-allocated jobs, to minimize wasted computation. Preemption does not cause the preempted jobs to fail, because Hadoop jobs tolerate losing tasks; it only makes them take longer to finish.

for more details on configuration and understanding on preemption please go through below link

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_yarn_resource_mgt/content/preemption.html

Mentor

@Sunile Manjee

i believe it's based on utilization and this is the property that controls it yarn.scheduler.fair.preemption.cluster-utilization-threshold

The utilization threshold after which preemption kicks in. The utilization is computed as the maximum ratio of usage to capacity among all resources. Defaults to 0.8f

https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/FairScheduler.html