question Re: yarn scheduler question in Support Questions

yarn scheduler question

aliyesami — Sun, 23 Apr 2017 20:41:37 GMT

I asked this simple question before but got no response , trying again .

if I set these values for the scheduler in YARN config , it breaks YARN , what am I doing wrong here?

yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.queues=Engineering,Marketing,Support
yarn.scheduler.capacity.root.Engineering.acl_administer_jobs=*
yarn.scheduler.capacity.root.Engineering.acl_submit_applications=*
yarn.scheduler.capacity.root.Engineering.capacity=100
yarn.scheduler.capacity.root.Engineering.maximum-capacity=100
yarn.scheduler.capacity.root.Engineering.state=RUNNING
yarn.scheduler.capacity.root.Engineering.user-limit-factor=1
yarn.scheduler.capacity.Engineering.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.Marketing.acl_administer_jobs=*
yarn.scheduler.capacity.root.Marketing.acl_submit_applications=*
yarn.scheduler.capacity.root.Marketing.capacity=100
yarn.scheduler.capacity.root.Marketing.maximum-capacity=100
yarn.scheduler.capacity.root.Marketing.state=RUNNING
yarn.scheduler.capacity.root.Marketing.user-limit-factor=1
yarn.scheduler.capacity.Marketing.minimum-user-limit-percent=100

Re: yarn scheduler question

nmaillard1 — Sun, 23 Apr 2017 20:44:09 GMT

Hello Sami

Without the logs and or error message this is left up to guess. A quick look seems to show that your are allocating more than the total amount of ressources and that you do not seem to have defined the support queue you listed.

Re: yarn scheduler question

gnovak — Sun, 18 Aug 2019 04:25:28 GMT

@Sami Ahmad

I tried your config and the ResourceManager's logs show error: "Illegal capacity of -1.0 for queue root.Support".

Support is defined in root.queues, but it's capacity is undefined. If you fixed that (add yarn.scheduler.capacity.root.Support.capacity) the next problem will be, that the sum of the capacities of any queue's child queues must be 100%. In your case, root.Marketing + root.Engineering = 200%. Try the following:

yarn.scheduler.capacity.root.Engineering.capacity=40
yarn.scheduler.capacity.root.Marketing.capacity=40
yarn.scheduler.capacity.root.Support.capacity=20

or any other 3 numbers that add up to 100.

If you fix this too, your config will be valid: