Support Questions
Find answers, ask questions, and share your expertise

CapacityScheduler configuration problem.

Explorer

Hdp version is 2.6.3 and ambari is the newest.

But I got some wrong information about this, could you please help me out of this ?

[Mon Jul 30 17:49:02 +0800 2018] Application is added to the scheduler and is not yet activated. User's AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:6144, vCores:1>; Queue Resource Limit for AM = <memory:106496, vCores:1>; User AM Resource Limit of the queue = <memory:75776, vCores:1>; Queue AM Resource Usage = <memory:75776, vCores:10>;

Below is my configuration for capacity Scheduler:

yarn.scheduler.capacity.maximum-am-resource-percent=0.1 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.root.accessible-node-labels=* yarn.scheduler.capacity.root.acl_administer_queue=* yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.default.acl_submit_applications=* yarn.scheduler.capacity.root.default.capacity=70 yarn.scheduler.capacity.root.default.maximum-capacity=100 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.user-limit-factor=1 yarn.scheduler.capacity.root.queues=default,oozie-launcher-queue yarn.scheduler.capacity.queue-mappings-override.enable=false yarn.scheduler.capacity.root.oozie-launcher-queue.acl_administer_queue=* yarn.scheduler.capacity.root.oozie-launcher-queue.acl_submit_applications=* yarn.scheduler.capacity.root.oozie-launcher-queue.capacity=30 yarn.scheduler.capacity.root.oozie-launcher-queue.maximum-capacity=90 yarn.scheduler.capacity.root.oozie-launcher-queue.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.oozie-launcher-queue.ordering-policy=fifo yarn.scheduler.capacity.root.oozie-launcher-queue.state=RUNNING yarn.scheduler.capacity.root.oozie-launcher-queue.user-limit-factor=1

Below is other configurations:

Memory allocated for all YARN containers on a node is 208G

The container min size is 2, max is 10G.

App mater size is :8G

I also found the resource is not fully used, it only used small part. Only 12 app is running,other is pending status.

Please help and thanks in advance.

2 REPLIES 2

Re: CapacityScheduler configuration problem.

Cloudera Employee

yarn.scheduler.capacity.maximum-am-resource-percent=0.1 could be the problem, you can try to increase it to like 0.4/0.5

And if you don't have many user concurrently using resource at the same time, yarn.scheduler.capacity.root.default.user-limit-factor=1 can be increased to sometime like 2 or 3 to get higher resource utilization.

Re: CapacityScheduler configuration problem.

Explorer

hi Wtan,

Thanks for your reply. I have tried it,It works fine.