Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn Fair scheduler: Applications within the queue should not kill each other

Yarn Fair scheduler: Applications within the queue should not kill each other

Explorer

Is there any way to switch off Applications "within" the queue should not kill each other, only preempt the applications of other queues.

 

We are on CDH 5.14, usign Dynamic pools. lets say 100 applications in the queue A, 10 applications are running, they should keep running even after Fairshare timeout, only after as soon as the containers finished from the 10 remaining jobs should be allocated. Preemption enabled becuase Quaue B, C can be active any time. If I put maimum applications per queue 10, cluster is under utilized.

 

Any thoughts?

5 REPLIES 5

Re: Yarn Fair scheduler: Applications within the queue should not kill each other

Cloudera Employee

An application will never get preempted for an application in the same queue. The preemption logic works only for prempting from another queue whihc is running above fairshare. So if 10 applications are running in Queue A  and taking all resource and queue A is running over fairshare , the a new application coming in Queure A will wait till some application finish and resource are available. The 10 application will not get preempted.

 

But if a job is posted in Queue B and preemption is enabled on Queue A ( and otehre conmdititons satisfied), the applications in Queue A can get preempted. 

 

So most probably your application is getting preempted due to other queue and not becasue of same queue applications.  You can disable preemption on critical queues so taht no otehr queue can also preempt from that critical queue.

Re: Yarn Fair scheduler: Applications within the queue should not kill each other

Explorer

hmm... I am confused.. can you take a look at following explanation

 

https://blog.cloudera.com/blog/2018/06/yarn-fairscheduler-preemption-deep-dive/

 

Distribution of FairShare to applications within a Queue

Applications running in a queue always evenly split their queue’s Instantaneous FairShare. While FairShare is configured at the queue level, for the purpose of FairShare starvation it is the application-level FairShare that matters. This is significant because one or more applications in a queue may be FairShare-starved even though the queue itself is not FairShare-starved. An application that exceeds its Instantaneous FairShare can have its containers preempted by other FairShare-starved applications in any queue. This could thus be an application running in the same queue also.

Re: Yarn Fair scheduler: Applications within the queue should not kill each other

Cloudera Employee

Sorry about the confusion, yes you are right, I completely missed the part that inside a queue application can be preempted. Thanks for pointing out the Engineering blog whihc elarly states that, otherwise I see lots of other documentation does not explicitly mentions this and talk [preemption only in terms of other queue. 

 

Now , Is there a way to switch off Applications "within" the queue: I went through all fair share configurations, there is no configuration to switch off preemption only for the within queue. You can switch preemption at the queue level. The workaround can be to create multiple queues and then use one queue for critical jobs whihc you do not want to get preempted and disbale preemption on that particular queue. Use other queues whihc you are ok with geting preempted.

In that case, even if you submit a new critical job to that queue, it will not preempt any container and should wait till previous submitted critical jobs releases some resoruce.

 

If I put maximum applications per queue 10, cluster is under utilized: Yes setting max application might not be good idea in this usecase. Even if set it to 10 , then if 2 application has take all resource , a third applciation can come and preempt those containers.

 

So for this use case I think splitting the queues and configuring individual queues as mentioned above. I am not able to think of any better solution. Will like to hear your feedback.

Highlighted

Re: Yarn Fair scheduler: Applications within the queue should not kill each other

Explorer

Thank you @Bimalc 

 

I am already doing which you suggested, I have this use case where I need disbale inter-queue-preemption

 

I think capacity scheduler code has atleast these words, but not fair scheduler.

 

I opened a upstream yarn Jira: https://issues.apache.org/jira/browse/YARN-9520

Re: Yarn Fair scheduler: Applications within the queue should not kill each other

Explorer

@Bimalc 

 

https://issues.apache.org/jira/browse/YARN-9520

 

The suggestion from the above JIRA is to use FIFO policy, just for queue A. I think its good idea, it will take care of inter-queue-preemption

 

Any nuances using FIFO policy on Fair scheduler?

Don't have an account?
Coming from Hortonworks? Activate your account here