06-07-2019 02:54 PM
Is a parent's instatanious fair share ever used in preemption calculations? Can a child/leaf queue process be preempted if the child/leaf queue is over its instatanious fair share even though the parent is not? This would make sense to preempt within a parent to give to another child, but not give to another non child queue.
In attached picture, parent not over but chhild is. Since parent not over I would think that its resources are only available to its own children, not others.
06-10-2019 11:31 PM
A parent's fair share is always used in preemption calculations because a child's fair share is derived from the parent. See YARN FairScheduler Preemption Deep Dive for more details around this.
Jobs can only be submitted to leaf queues. If a child queue is over its instantaneous fair share even if parent is not, then, it is possible that a container in this queue can get preempted.
It seems that the parent queue you are referring to from the screenshot is root.bis and the child queue that is over its fair share is root.bis.edf_bis_sbfe_qa. Before we get into more details, I would like to highlight the fact that this screenshot from the RM Web UI's scheduler page may not be taking vcores into account i.e. this page only accounts for fair shares based on memory. You may have a completely different picture with regards to vcores, as such, you really need to look at the Fair Scheduler dump from the Resource Manager to understand which queues if any were over their fair share. Moving on to your question, the generic answer with regards to resource redistribution after preemption is that the freed resources may be made available to any starved application of another queue that may or may not be part of the same parent or hierarchy. There is no setting to influence that either.