question YARN capacity confiurations Illegal capacity of 0.25 for children of queue root in Support Questions

YARN capacity confiurations Illegal capacity of 0.25 for children of queue root

daniel_hopkins — Fri, 16 Sep 2022 14:10:27 GMT

I am trying to adjust the yarn capacity settings..for a newly setup hive/druid stack.

The system was 'working' but then this week it started complaining about

Failed to submit application_1550602511762_0005 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 0 applications, cannot accept submission of application: application_1550602511762_0005

it was suggested I needed to reduce the capacity of the llap queue.

my 'scheduling' config I am trying..

yarn.scheduler.capacity.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_administer_jobs=*
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=0.0
yarn.scheduler.capacity.root.default.maximum-capacity=0.0
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=1
yarn.scheduler.capacity.root.llap.acl_administer_queue=hive
yarn.scheduler.capacity.root.llap.acl_submit_applications=hive
yarn.scheduler.capacity.root.llap.capacity=25.0
yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.llap.maximum-capacity=100.0
yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.ordering-policy=fifo
yarn.scheduler.capacity.root.llap.state=RUNNING
yarn.scheduler.capacity.root.llap.user-limit-factor=1
yarn.scheduler.capacity.root.queues=default,llap
yarn.scheduler.capacity.schedule-asynchronously.enable=true
yarn.scheduler.capacity.schedule-asynchronously.maximum-threads=1
yarn.scheduler.capacity.schedule-asynchronously.scheduling-interval-ms=10

caused by: java.io.IOException: Failed to re-init queues : Illegal capacity of 0.25 for children of queue root    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:478)    
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)    
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:394)    
.. 10 moreCaused by: java.lang.IllegalArgumentException: Illegal capacity of 0.25 for children of queue roots

I have tried various configs..that add up to 100 or are less than 100, but it keeps complaining..

Re: YARN capacity confiurations Illegal capacity of 0.25 for children of queue root

jsensharma — Wed, 20 Feb 2019 10:49:25 GMT

@Dan Hops

Can you try changing the

"yarn.scheduler.capacity.root.default.capacity=0.0" to "yarn.scheduler.capacity.root.default.capacity=75"

"yarn.scheduler.capacity.root.default.maximum-capacity=0.0" to "yarn.scheduler.capacity.root.default.maximum-capacity=100"

Similarly change the "yarn.scheduler.capacity.root.llap.capacity=25.0" to "yarn.scheduler.capacity.root.llap.capacity=25"

Example:

yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=75
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=1
yarn.scheduler.capacity.root.queues=default,llap
yarn.scheduler.capacity.queue-mappings-override.enable=false
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.default.priority=0
yarn.scheduler.capacity.root.llap.acl_administer_queue=hive
yarn.scheduler.capacity.root.llap.acl_submit_applications=hive
yarn.scheduler.capacity.root.llap.capacity=25
yarn.scheduler.capacity.root.llap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.llap.maximum-capacity=100
yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.llap.ordering-policy=fifo
yarn.scheduler.capacity.root.llap.priority=0
yarn.scheduler.capacity.root.llap.state=RUNNING
yarn.scheduler.capacity.root.llap.user-limit-factor=1
yarn.scheduler.capacity.root.priority=0

Re: YARN capacity confiurations Illegal capacity of 0.25 for children of queue root

daniel_hopkins — Thu, 21 Feb 2019 00:52:50 GMT

that seems to work no error in the log, the issue was that I had it specified in 'floating point'?

Also now it my 'job' is accepted on the applications page, but it never starts..

There is one other job running on that queue (default), however when I try to kill it..it just errors out..with Error: Kill application failed!

Re: YARN capacity confiurations Illegal capacity of 0.25 for children of queue root

daniel_hopkins — Thu, 21 Feb 2019 00:58:57 GMT

ON the yarn queues tab, I see: root (100%), default(0%), and llap(100%)..does this may changes had no effect??