In our environment we use capacity scheduler. Pre-emption is disabled. I came across a situation i could not understand and would appreciate any insights from the forum members. let us suppose i have four queues in my cluster. q1 to q4. The capacity allocation for q1, q2 and q3 is 30 minimum and 50 maximum. The capacity for q4 is 10 minimum and 100 maximum. So we have jobs running on all queues. But only some 60% of the cluster capacity is being used currently. One running job currently uses the q4 queue and the usage is over capacity at 200% ie 20% of the cluster capacity. Another job for q4 is executed, but goes into accepted state. So the question is even though q4 has a max capacity of 100% and the current usage is only 60% of the capacity of the cluster, still it is not allowing another process to execute on the same queue. This is the issue I saw. So is the new job not able to execute in q4 queue because q4 is already 'over capacity'? Or is it any other reason? If the former, then can only current running jobs in q4 avail of the 100%? Or can the capacity allocation for the queues be designed better? Appreciate the insights. ... View more

under yarn you can configure total space allocation for containers, minimum container size and maximum container size. this container configuration seems to be linked with the size of the memory on nodes. so the total space allocation for containers is always less than the total memory on the hosts? but a container is not only memory right? it is also processor cores, disk space, network bandwidth etc. so where is the accommodation for the rest of these resources (cores, disks etc) in the container sizing? appreciate the insights. ... View more