I have 4 queues under the root queue with the following configuration.
Capacity (in %)
Max Capacity (in %)
User Limit Factor
I have set up capacity by yarn.scheduler.capacity.root.<queue-name>.capacity and max capacity by yarn.scheduler.capacity.root.<queue-name>.maximum-capacity property.
My understanding is, above 2 properties set ABSOLUTE capacity and ABSOLUTE maximum capacity that means queue stream's 100% is equal to the 70% of cluster's total capacity and it can fill up to 100% of queue's capacity that is also the 70% of cluster's total capacity.
Now, the problem is when queue 'stream' is filled with 66.4% (i.e. when Used Capacity: 66.4% & Absolute Used Capacity: 46.5%) then new jobs are getting in the pending state which is submitted in queue 'stream' by saying "waiting for AM container to be allocated, launched and register with RM".
When I checked queue configuration on yarn UI it shows Configured Max Capacity: 70.0% & Absolute Configured Max Capacity: 70.0% but according to the configuration, queue 'stream' can be filled till Used Capacity: 100% & Absolute Used Capacity: 70%
Any idea, why new jobs are unable to utilize the queue stream's capacity till 100%?