Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn Preemption Parent Leaf Queues Fair Share Scheduler

Yarn Preemption Parent Leaf Queues Fair Share Scheduler

Explorer

Is a parent's instatanious fair share ever used in preemption calculations? Can a child/leaf queue process be preempted if the child/leaf queue is over its instatanious fair share even though the parent is not? This would make sense to preempt within a parent to give to another child, but not give to another non child queue.

 

In attached picture, parent not over but chhild is. Since parent not over I would think that its resources are only available to its own children, not others.


Capture.PNG
1 REPLY 1
Highlighted

Re: Yarn Preemption Parent Leaf Queues Fair Share Scheduler

Cloudera Employee

@regeamor 

 

A parent's fair share is always used in preemption calculations because a child's fair share is derived from the parent. See YARN FairScheduler Preemption Deep Dive for more details around this.


Jobs can only be submitted to leaf queues. If a child queue is over its instantaneous fair share even if parent is not, then, it is possible that a container in this queue can get preempted.

 

It seems that the parent queue you are referring to from the screenshot is root.bis and the child queue that is over its fair share is root.bis.edf_bis_sbfe_qa. Before we get into more details, I would like to highlight the fact that this screenshot from the RM Web UI's scheduler page may not be taking vcores into account i.e. this page only accounts for fair shares based on memory. You may have a completely different picture with regards to vcores, as such, you really need to look at the Fair Scheduler dump from the Resource Manager to understand which queues if any were over their fair share. Moving on to your question, the generic answer with regards to resource redistribution after preemption is that the freed resources may be made available to any starved application of another queue that may or may not be part of the same parent or hierarchy. There is no setting to influence that either.

Don't have an account?
Coming from Hortonworks? Activate your account here