Created 01-13-2016 05:37 AM
@Ram D you have an oversubscribed queue, it says you have a total of 14GB of RAM on your cluster with 14GB used, so until the first job finishes, you won't have any progress. Easy test is to kill the first job and you can see the 2nd job go from accepted to running.
Created 02-02-2016 06:52 PM
Capacity scheduler must have default partition for AM container calculation. We added two more nodes as default partition, now it is accepting multiple jobs and working fine. AM has a bug in 2.7, without default partition we can't implement capacity-scheduler to make use of entire cluster. Without default partition, only node resources are available as entire cluster resources. So, it is unable to run the second job, remain in accepted state.
Created 02-02-2016 06:54 PM
Created 02-02-2016 07:21 PM
Created 12-31-2017 02:48 AM
I could see that you have mentioned the warning in the log was -
"not starting application as amIfStarted exceeds amLimit"
This indicates that the yarnClientApplication request to launch the ApplicationMaster container is not able to get the resource in the current queue, where the AM container is being tried to be launched. The Maximum Percentage of AM containers have reached threshold in the current queue. You can wait for the currently running AM container to complete or try to increase the threshold.
I would suggest to first take a quick look on the "maxAMResourcePerQueuePercent" for the queue where this ApplicationMaster is launched. This can be checked from the Capacity Scheduler configuration in Yarn. Try increasing the capacity of "maxAMResourcePerQueuePercent" to a higher value. This way you are increasing the threshold and are allowing more AM container to be run in the current queue.