Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

CapacityScheduler configuration problem.

Explorer

Hdp version is 2.6.3 and ambari is the newest.

But I got some wrong information about this, could you please help me out of this ?

[Mon Jul 30 17:49:02 +0800 2018] Application is added to the scheduler and is not yet activated. User's AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:6144, vCores:1>; Queue Resource Limit for AM = <memory:106496, vCores:1>; User AM Resource Limit of the queue = <memory:75776, vCores:1>; Queue AM Resource Usage = <memory:75776, vCores:10>;

Below is my configuration for capacity Scheduler:

yarn.scheduler.capacity.maximum-am-resource-percent=0.1 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.root.accessible-node-labels=* yarn.scheduler.capacity.root.acl_administer_queue=* yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.default.acl_submit_applications=* yarn.scheduler.capacity.root.default.capacity=70 yarn.scheduler.capacity.root.default.maximum-capacity=100 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.user-limit-factor=1 yarn.scheduler.capacity.root.queues=default,oozie-launcher-queue yarn.scheduler.capacity.queue-mappings-override.enable=false yarn.scheduler.capacity.root.oozie-launcher-queue.acl_administer_queue=* yarn.scheduler.capacity.root.oozie-launcher-queue.acl_submit_applications=* yarn.scheduler.capacity.root.oozie-launcher-queue.capacity=30 yarn.scheduler.capacity.root.oozie-launcher-queue.maximum-capacity=90 yarn.scheduler.capacity.root.oozie-launcher-queue.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.oozie-launcher-queue.ordering-policy=fifo yarn.scheduler.capacity.root.oozie-launcher-queue.state=RUNNING yarn.scheduler.capacity.root.oozie-launcher-queue.user-limit-factor=1

Below is other configurations:

Memory allocated for all YARN containers on a node is 208G

The container min size is 2, max is 10G.

App mater size is :8G

I also found the resource is not fully used, it only used small part. Only 12 app is running,other is pending status.

Please help and thanks in advance.

2 REPLIES 2

Cloudera Employee

yarn.scheduler.capacity.maximum-am-resource-percent=0.1 could be the problem, you can try to increase it to like 0.4/0.5

And if you don't have many user concurrently using resource at the same time, yarn.scheduler.capacity.root.default.user-limit-factor=1 can be increased to sometime like 2 or 3 to get higher resource utilization.

Explorer

hi Wtan,

Thanks for your reply. I have tried it,It works fine.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.