Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

YARN capacity confiurations Illegal capacity of 0.25 for children of queue root

avatar
Contributor

I am trying to adjust the yarn capacity settings..for a newly setup hive/druid stack.

The system was 'working' but then this week it started complaining about

Failed to submit application_1550602511762_0005 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 0 applications, cannot accept submission of application: application_1550602511762_0005

it was suggested I needed to reduce the capacity of the llap queue.

my 'scheduling' config I am trying..

yarn.scheduler.capacity.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_administer_jobs=*
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=0.0
yarn.scheduler.capacity.root.default.maximum-capacity=0.0
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=1
yarn.scheduler.capacity.root.llap.acl_administer_queue=hive
yarn.scheduler.capacity.root.llap.acl_submit_applications=hive
yarn.scheduler.capacity.root.llap.capacity=25.0
yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.llap.maximum-capacity=100.0
yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.ordering-policy=fifo
yarn.scheduler.capacity.root.llap.state=RUNNING
yarn.scheduler.capacity.root.llap.user-limit-factor=1
yarn.scheduler.capacity.root.queues=default,llap
yarn.scheduler.capacity.schedule-asynchronously.enable=true
yarn.scheduler.capacity.schedule-asynchronously.maximum-threads=1
yarn.scheduler.capacity.schedule-asynchronously.scheduling-interval-ms=10



caused by: java.io.IOException: Failed to re-init queues : Illegal capacity of 0.25 for children of queue root    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:478)    
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)    
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:394)    
.. 10 moreCaused by: java.lang.IllegalArgumentException: Illegal capacity of 0.25 for children of queue roots

I have tried various configs..that add up to 100 or are less than 100, but it keeps complaining..



1 ACCEPTED SOLUTION

avatar
Master Mentor

@Dan Hops

Can you try changing the

"yarn.scheduler.capacity.root.default.capacity=0.0" to "yarn.scheduler.capacity.root.default.capacity=75"

"yarn.scheduler.capacity.root.default.maximum-capacity=0.0" to "yarn.scheduler.capacity.root.default.maximum-capacity=100"

Similarly change the "yarn.scheduler.capacity.root.llap.capacity=25.0" to "yarn.scheduler.capacity.root.llap.capacity=25"

Example:

yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=75
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=1
yarn.scheduler.capacity.root.queues=default,llap
yarn.scheduler.capacity.queue-mappings-override.enable=false
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.default.priority=0
yarn.scheduler.capacity.root.llap.acl_administer_queue=hive
yarn.scheduler.capacity.root.llap.acl_submit_applications=hive
yarn.scheduler.capacity.root.llap.capacity=25
yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.llap.maximum-capacity=100
yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.ordering-policy=fifo
yarn.scheduler.capacity.root.llap.priority=0
yarn.scheduler.capacity.root.llap.state=RUNNING
yarn.scheduler.capacity.root.llap.user-limit-factor=1
yarn.scheduler.capacity.root.priority=0


View solution in original post

3 REPLIES 3

avatar
Master Mentor

@Dan Hops

Can you try changing the

"yarn.scheduler.capacity.root.default.capacity=0.0" to "yarn.scheduler.capacity.root.default.capacity=75"

"yarn.scheduler.capacity.root.default.maximum-capacity=0.0" to "yarn.scheduler.capacity.root.default.maximum-capacity=100"

Similarly change the "yarn.scheduler.capacity.root.llap.capacity=25.0" to "yarn.scheduler.capacity.root.llap.capacity=25"

Example:

yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=75
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=1
yarn.scheduler.capacity.root.queues=default,llap
yarn.scheduler.capacity.queue-mappings-override.enable=false
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.default.priority=0
yarn.scheduler.capacity.root.llap.acl_administer_queue=hive
yarn.scheduler.capacity.root.llap.acl_submit_applications=hive
yarn.scheduler.capacity.root.llap.capacity=25
yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.llap.maximum-capacity=100
yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.ordering-policy=fifo
yarn.scheduler.capacity.root.llap.priority=0
yarn.scheduler.capacity.root.llap.state=RUNNING
yarn.scheduler.capacity.root.llap.user-limit-factor=1
yarn.scheduler.capacity.root.priority=0


avatar
Contributor

that seems to work no error in the log, the issue was that I had it specified in 'floating point'?

Also now it my 'job' is accepted on the applications page, but it never starts..

There is one other job running on that queue (default), however when I try to kill it..it just errors out..with Error: Kill application failed!



avatar
Contributor

ON the yarn queues tab, I see: root (100%), default(0%), and llap(100%)..does this may changes had no effect??