Yarn Capacity Scheduler: Share resource between queues

[#1]

Hi All,

I research a lot but not clearly about Share resource between Yarn queues

My question is.

I have two queue:

- Dev:

Capacity: 65%

Max Capacity: 100%

User limit factor: 1

- Test:

Capacity: 35%

Max Capacity: 100%

User limit factor: 1

I want If job on queue Test not using all resource, job on Dev can using free resource on Test. And it do the same with job on queue Test if queue Dev have free resource.

[#2 · Accepted solution]

@Hoang Le

Capacity Scheduler’s leaf queues can also use the user-limit-factor property to control user resource allocations. This property denotes the fraction of queue capacity that any single user can consume up to a maximum value, regardless of whether or not there are idle resources in the cluster.

Property: yarn.scheduler.capacity.root.support.user-limit-factor

Value: 1

The default value of "1" means that any single user in the queue can at maximum only occupy the queue’s configured capacity. This prevents users in a single queue from monopolizing resources across all queues in a cluster. Setting the value to "2" would restrict the queue's users to twice the queue’s configured capacity. Setting it to a value of 0.5 would restrict any user from using resources beyond half of the queue capacity.

These settings can also be dynamically changed at run-time using yarn rmadmin - refreshQueues.

Please change user limit factor and try.