Preemption configuration

[#1]

Issue:

After enabling the preemption parameters in yarn-site.xml as per link, it was found the applications are still waiting to run. This is because other queues are utilising all of the available resources. With Preemption enabled, under-served queues can begin to claim their allocated cluster resources almost immediately, without having to wait for other queues' applications to finish running.

Resolution:

Along with the below list of parameters to enable preemption, configure this parameter "yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity" in yarn-site.xml, otherwise you won't see preemption though enabled. The doc is missing the above parameter.

List of parameters to enable preemption are:

1) yarn.resourcemanager.scheduler.monitor.enable

2) yarn.resourcemanager.scheduler.monitor.policies

3) yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval

4) yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill

5) yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round

6) yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor

Reference:

https://hortonworks.com/blog/better-slas-via-resource-preemption-in-yarns-capacityscheduler/

Addendum:

Logged internal bug to include this parameter in Hortonworks official documentation