spark-shell not getting launched - Queue's AM resource limit exceeded.

[#1]

Hello - i've HDP 2.5.x and i'm trying to launch spark-shell .. ApplicationMaster gets launched, but YARN is not able to assign containers.

Command ->

./bin/spark-shell --master yarn-client --driver-memory 512m --executor-memory 512m

Error ->

[Sun Aug 06 19:33:29 +0000 2017] Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:2048, vCores:1>; Queue Resource Limit for AM = <memory:6144, vCores:1>; User AM Resource Limit of the queue = <memory:6144, vCores:1>; Queue AM Resource Usage = <memory:6144, vCores:3>;

Any ideas on what parameters to change ?

Pls note -> In YARN, the parameter - yarn.scheduler.capacity.maximum-am-resource-percent = 0.9 , AM should have access sufficient to assign container

[#2]

@smanjee, @mqureshi, @Neeraj Sabharwal, - any ideas on this ?

[#3]

This is what i see in the Ambari (YARN capacity scheduer config)

---------------

yarn.scheduler.capacity.root.queues=default,llap
yarn.scheduler.capacity.root.default.user-limit-factor=1 
yarn.scheduler.capacity.root.default.state=RUNNING 
yarn.scheduler.capacity.root.default.maximum-capacity=60 
yarn.scheduler.capacity.root.default.capacity=60 
yarn.scheduler.capacity.root.default.acl_submit_applications=* 
yarn.scheduler.capacity.root.default.acl_administer_jobs=* 
yarn.scheduler.capacity.root.capacity=100 
yarn.scheduler.capacity.root.acl_administer_queue=* 
yarn.scheduler.capacity.root.accessible-node-labels=* 
yarn.scheduler.capacity.node-locality-delay=40 
yarn.scheduler.capacity.maximum-applications=10000 
yarn.scheduler.capacity.maximum-am-resource-percent=0.9 
yarn.scheduler.capacity.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.acl_administer_queue=hive
yarn.scheduler.capacity.root.llap.acl_submit_applications=hive
yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.llap.maximum-capacity=40
yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.ordering-policy=fifo
yarn.scheduler.capacity.root.llap.state=RUNNING
yarn.scheduler.capacity.root.llap.user-limit-factor=1
yarn.scheduler.capacity.root.llap.capacity=40

[#4]

@Karan Alang

Could you please share (at least the relevant part of) your Capacity Scheduler config and tell us how much memory your default queue in total should have? Based on your error your default queue's AM limit is in fact exceeded..

[#5]

@gnovak - here is what i see in YARN Capacity Scheduler

yarn.scheduler.capacity.root.queues=default,llap
yarn.scheduler.capacity.root.default.user-limit-factor=1 
yarn.scheduler.capacity.root.default.state=RUNNING 
yarn.scheduler.capacity.root.default.maximum-capacity=60 
yarn.scheduler.capacity.root.default.capacity=60 
yarn.scheduler.capacity.root.default.acl_submit_applications=* 
yarn.scheduler.capacity.root.default.acl_administer_jobs=* 
yarn.scheduler.capacity.root.capacity=100 
yarn.scheduler.capacity.root.acl_administer_queue=* 
yarn.scheduler.capacity.root.accessible-node-labels=* 
yarn.scheduler.capacity.node-locality-delay=40 
yarn.scheduler.capacity.maximum-applications=10000 
yarn.scheduler.capacity.maximum-am-resource-percent=0.9 
yarn.scheduler.capacity.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.acl_administer_queue=hive
yarn.scheduler.capacity.root.llap.acl_submit_applications=hive
yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.llap.maximum-capacity=40
yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.ordering-policy=fifo
yarn.scheduler.capacity.root.llap.state=RUNNING
yarn.scheduler.capacity.root.llap.user-limit-factor=1 
yarn.scheduler.capacity.root.llap.capacity=40 

[#6]

@Karan Alang

Based on this information I assume you have 12 GBs of memory and the minimum allocation is set to 1024 MB, the default queue has a configured capacity of 60%, 7 GBs. The AM limit is 6 GBs (7.2 * 0.9 rounded to GB), and it is full, probably three other AMs are running. Please correct me if I'm wrong!

To get more memory, you might try these things:

• Add more memory to the cluster 😛
• Increase the maximum-capacity of the default queue, so that it can use more resources when the LLAP queue doesn't use them
• Increase the maximum-am-resource-percent of the default queue to 1
• Decrease the minimum-allocation-mb: this way the other AMs (and containers) might use less resources (e.g. if you need 1.2 GBs - just for the sake of the example - then with the default 1 GB minimum allocation you still need to get a 2 GB container)
• Kill other applications from the queue or wait until they finish

[#7]

@gnovak - Thanks, how did you conclude that my cluster has 12 GB RAM .. i mean - what parameter indicates that ?

[#8]

The default queue's AM limit is 6144 MB -> the default queue's capacity must be 7 GB (for 6 the limit would be 5 and for 8 it would be 7 with the maximum-am-resource-percent of 0.9). Since default.capacity = 60 the whole cluster's capacity equals to ~ 100 / 60 * 7, which could indicate 12 or 13 GBs in total, but the latter would be very unusual.

Did you manage to overcome your issue with any of my suggestions?