e.g. Consider below scenario
1. I have 2 queues configured viz. A and B
2. For Queue A, I have 40% allocated capacity and maximum allocation is set to 100%.
3. For Queue B, I have 60% allocated capacity and Max is set to 80%
4. I have submitted heavy job in queue A
5. When I did step 4, queue B was all empty ( no jobs were running )
6. Because of elasticity feature, RM will allocate resources from queue B to queue A as max allocation for A is set to 100% and B queue is empty
7. Now user submitted job in queue B which was empty before
8. RM will inform AM to free up resources from queue A and to do so, pre-emption will happen.
To reduce pre-emption:
I can reduce maximum allocation to lower value like 50% or increase allocated capacity
To disable pre-emption:
I can disable it from yarn configs however then for above situation, job in queue B will starve until heavy job in queue A gets completed.
Third option I can think of is:
To reserve x amount of containers for an application so that it will always get those reserved resources and pre-emption will never happen. is it possible at all? if Yes, how to do that.
Please correct me if my understanding is wrong, any other approach is appreciated.
If it is 2 queues, you can already achieve that with your maximum allocation. If your Queue A max allocation is 80% and you have only 2 queues, then you are already reserving 20% for Queue B.
But if you have more than two queues, this is not possible. Is there a reason why you don't want to use pre-emption in that case? We had a case where there are certain type of jobs that does not work well under preemption (long running jobs). In that case, we ended up using allocated and maximum to same % for that queue, but leave preemption on across the cluster. This would mean, jobs in that queue are not preempted but jobs in other queues can be preempted.
Actually there is not really a difference between simply not setting max capacity higher than allocated capacity.
If you want to have two queues that take resources from each other but you want to have a third queue that should have a guaranteed set you could use nested queues
A_GUARANTEED = allocated 30% Max 100%
B_EVERYTHINGELSE = allocated 70% MAX 70%
B_EVERYTHINGELSE.queue1 = allocated 50% max 100%
B_EVERYTHINGELSE.queue2 = allocated 50% max 100%
Unless I am completely mistaken you then have the desired result 30% of the cluster will always be reserved for queue A and the other queues fight it out for the remaining 70%