Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CapacityScheduler configuration problem.

CapacityScheduler configuration problem.

Explorer

Hdp version is 2.6.3 and ambari is the newest.

But I got some wrong information about this, could you please help me out of this ?

[Mon Jul 30 17:49:02 +0800 2018] Application is added to the scheduler and is not yet activated. User's AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:6144, vCores:1>; Queue Resource Limit for AM = <memory:106496, vCores:1>; User AM Resource Limit of the queue = <memory:75776, vCores:1>; Queue AM Resource Usage = <memory:75776, vCores:10>;

Below is my configuration for capacity Scheduler:

yarn.scheduler.capacity.maximum-am-resource-percent=0.1 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.root.accessible-node-labels=* yarn.scheduler.capacity.root.acl_administer_queue=* yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.default.acl_submit_applications=* yarn.scheduler.capacity.root.default.capacity=70 yarn.scheduler.capacity.root.default.maximum-capacity=100 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.user-limit-factor=1 yarn.scheduler.capacity.root.queues=default,oozie-launcher-queue yarn.scheduler.capacity.queue-mappings-override.enable=false yarn.scheduler.capacity.root.oozie-launcher-queue.acl_administer_queue=* yarn.scheduler.capacity.root.oozie-launcher-queue.acl_submit_applications=* yarn.scheduler.capacity.root.oozie-launcher-queue.capacity=30 yarn.scheduler.capacity.root.oozie-launcher-queue.maximum-capacity=90 yarn.scheduler.capacity.root.oozie-launcher-queue.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.oozie-launcher-queue.ordering-policy=fifo yarn.scheduler.capacity.root.oozie-launcher-queue.state=RUNNING yarn.scheduler.capacity.root.oozie-launcher-queue.user-limit-factor=1

Below is other configurations:

Memory allocated for all YARN containers on a node is 208G

The container min size is 2, max is 10G.

App mater size is :8G

I also found the resource is not fully used, it only used small part. Only 12 app is running,other is pending status.

Please help and thanks in advance.

2 REPLIES 2
Highlighted

Re: CapacityScheduler configuration problem.

New Contributor

yarn.scheduler.capacity.maximum-am-resource-percent=0.1 could be the problem, you can try to increase it to like 0.4/0.5

And if you don't have many user concurrently using resource at the same time, yarn.scheduler.capacity.root.default.user-limit-factor=1 can be increased to sometime like 2 or 3 to get higher resource utilization.

Highlighted

Re: CapacityScheduler configuration problem.

Explorer

hi Wtan,

Thanks for your reply. I have tried it,It works fine.

Don't have an account?
Coming from Hortonworks? Activate your account here