YARN scheduler regarding intra queue resource allocation

[#1]

I have a bit confusion regarding these two properties, Minimum user limit : which specifies minimum resource a user is guaranteed and Ordering policy: fifo and fairNow if I set Minimum user limit to 100 %, then how will the second property come into picture if is set to "fair".

Also if i set the Minimum user limit to 50 % and, user1 job is utilizing 100 % of cluster resource, then user2 submit job who requires 20 % of cluster resource then will the resource get distributed as 80% and 20% or will it be 50% - 50%

[#2 · Accepted solution]

@btandel

1) Minimum user limit percentage, this is the definition from the documentation, I think this is as clear as it gets:

"Each queue enforces a limit on the percentage of resources allocated to a user at any given time, if there is demand for resources. The user limit can vary between a minimum and maximum value. The former (the minimum value) is set to this property value and the latter (the maximum value) depends on the number of users who have submitted applications. For e.g., suppose the value of this property is 25. If two users have submitted applications to a queue, no single user can use more than 50% of the queue resources. If a third user submits an application, no single user can use more than 33% of the queue resources. With 4 or more users, no user can use more than 25% of the queues resources. A value of 100 implies no user limits are imposed. The default is 100. Value is specified as a integer."

2) Fair ordering policy. Check this documentation: Using Flexible Scheduling Policies

The two both concerns a single queue's scheduling policy: minimum-user-limit-percentage defines how the queue's resources are distributed among users and the ordering policy defines in which order the submitted jobs will be executed. If the minimum user limit is 100%, it means that there are no actual limits in place, so the fair ordering policy will do its best to give all the jobs "fair" amount of resources.

3) "if i set the Minimum user limit to 50 % and, user1 job is utilizing 100 % of cluster resource, then user2 submit job who requires 20 % of cluster resource then will the resource get distributed as 80% and 20% or will it be 50% - 50%"

It will be 80-20%, because user2 doesn't need any more resources. If they needed let's say 60%, or more, than the distribution would be 50-50%.

[#3]

@gnovak

Thanks a lot for the response, just one last query. So for proper resource sharing within one queue, it make sense to enable fair scheduling with enabling size-based-weight ?

[#4]

@btandel

It's not a must, usually the default (FIFO) scheduling policy works fine, because in a usual use case you need to be "fair" (in a sense) among the queues, not within one queue. But if you need equal resource sharing within one queue, it does make perfect sense.