Hi,
Here are some stats before the question:
From Ambari -> YARN -> Stats:
Number of available containers : 1000+
Number of YARN applications running : 22 ( out of which 18 are Hive-Tez based)
Number of allocated containers : 780
Number of containers pending for allocation : 12000+
My question is : When we have queue burst enabled up to 100% of cluster capacity, Why will Resource Manager not allocate containers to the pending/running job? What could be the reason that we see huge number of pending allocation ?
Is it that Tez calcuates the number of containers required for completion of the job and adds that to the pending container allocation and it gradually gets allocated when required ( like reduce stage's 3000 containers are blocked on map)?
Can any one enlighten please?