Is there any way to switch off Applications "within" the queue should not kill each other, only preempt the applications of other queues.
We are on CDH 5.14, usign Dynamic pools. lets say 100 applications in the queue A, 10 applications are running, they should keep running even after Fairshare timeout, only after as soon as the containers finished from the 10 remaining jobs should be allocated. Preemption enabled becuase Quaue B, C can be active any time. If I put maimum applications per queue 10, cluster is under utilized.
An application will never get preempted for an application in the same queue. The preemption logic works only for prempting from another queue whihc is running above fairshare. So if 10 applications are running in Queue A and taking all resource and queue A is running over fairshare , the a new application coming in Queure A will wait till some application finish and resource are available. The 10 application will not get preempted.
But if a job is posted in Queue B and preemption is enabled on Queue A ( and otehre conmdititons satisfied), the applications in Queue A can get preempted.
So most probably your application is getting preempted due to other queue and not becasue of same queue applications. You can disable preemption on critical queues so taht no otehr queue can also preempt from that critical queue.
hmm... I am confused.. can you take a look at following explanation
Distribution of FairShare to applications within a Queue
Applications running in a queue always evenly split their queue’s Instantaneous FairShare. While FairShare is configured at the queue level, for the purpose of FairShare starvation it is the application-level FairShare that matters. This is significant because one or more applications in a queue may be FairShare-starved even though the queue itself is not FairShare-starved. An application that exceeds its Instantaneous FairShare can have its containers preempted by other FairShare-starved applications in any queue. This could thus be an application running in the same queue also.
Sorry about the confusion, yes you are right, I completely missed the part that inside a queue application can be preempted. Thanks for pointing out the Engineering blog whihc elarly states that, otherwise I see lots of other documentation does not explicitly mentions this and talk [preemption only in terms of other queue.
Now , Is there a way to switch off Applications "within" the queue: I went through all fair share configurations, there is no configuration to switch off preemption only for the within queue. You can switch preemption at the queue level. The workaround can be to create multiple queues and then use one queue for critical jobs whihc you do not want to get preempted and disbale preemption on that particular queue. Use other queues whihc you are ok with geting preempted.
In that case, even if you submit a new critical job to that queue, it will not preempt any container and should wait till previous submitted critical jobs releases some resoruce.
If I put maximum applications per queue 10, cluster is under utilized: Yes setting max application might not be good idea in this usecase. Even if set it to 10 , then if 2 application has take all resource , a third applciation can come and preempt those containers.
So for this use case I think splitting the queues and configuring individual queues as mentioned above. I am not able to think of any better solution. Will like to hear your feedback.