@Priyan S Maybe it is because you did not set up pre-emption on Yarn? Without pre-emption the order in which the jobs were submitted to Q1,Q2 and Q41 is determining the capacity allocations. It may be that since the jobs on Q1 and Q2 were submitted first, they both grabbed the max allocation of their respective queues (40%). When the job for Q41 comes along there is just no more than the remaining 20% for Q4 and/or 100% for any of its subleafs Q41,42,43,44. I don't get why Q41 is only getting 10% and not 20%. You can look upon pre-emption as a way to help restore the state in which all queues get at least their minimum configured allocation, even though the missing part for one queue A. operating under its minimum might be used by another queue B. operating above its minimum (since it grabbed the excess capacity, up to its maximum, because it was still available at that time). Without pre-emption queue A. would have to wait for queue B. to release capacity of finished jobs in B. With pre-emption Yarn will actively free up resources of B. to allocate to A. in the process it might even kill job parts in queue B. to do so.
... View more
