Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn Capacity Scheduler: Share resource between queues

Solved Go to solution
Highlighted

Yarn Capacity Scheduler: Share resource between queues

Hi All,

I research a lot but not clearly about Share resource between Yarn queues

My question is.

I have two queue:

- Dev:

Capacity: 65%

Max Capacity: 100%

User limit factor: 1

- Test:

Capacity: 35%

Max Capacity: 100%

User limit factor: 1

I want If job on queue Test not using all resource, job on Dev can using free resource on Test. And it do the same with job on queue Test if queue Dev have free resource.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Yarn Capacity Scheduler: Share resource between queues

@Hoang Le

Capacity Scheduler’s leaf queues can also use the user-limit-factor property to control user resource allocations. This property denotes the fraction of queue capacity that any single user can consume up to a maximum value, regardless of whether or not there are idle resources in the cluster.

Property: yarn.scheduler.capacity.root.support.user-limit-factor

Value: 1

The default value of "1" means that any single user in the queue can at maximum only occupy the queue’s configured capacity. This prevents users in a single queue from monopolizing resources across all queues in a cluster. Setting the value to "2" would restrict the queue's users to twice the queue’s configured capacity. Setting it to a value of 0.5 would restrict any user from using resources beyond half of the queue capacity.

These settings can also be dynamically changed at run-time using yarn rmadmin - refreshQueues.

Please change user limit factor and try.

View solution in original post

1 REPLY 1

Re: Yarn Capacity Scheduler: Share resource between queues

@Hoang Le

Capacity Scheduler’s leaf queues can also use the user-limit-factor property to control user resource allocations. This property denotes the fraction of queue capacity that any single user can consume up to a maximum value, regardless of whether or not there are idle resources in the cluster.

Property: yarn.scheduler.capacity.root.support.user-limit-factor

Value: 1

The default value of "1" means that any single user in the queue can at maximum only occupy the queue’s configured capacity. This prevents users in a single queue from monopolizing resources across all queues in a cluster. Setting the value to "2" would restrict the queue's users to twice the queue’s configured capacity. Setting it to a value of 0.5 would restrict any user from using resources beyond half of the queue capacity.

These settings can also be dynamically changed at run-time using yarn rmadmin - refreshQueues.

Please change user limit factor and try.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here