yarn scheduler question

[#1]

I asked this simple question before but got no response , trying again .

if I set these values for the scheduler in YARN config , it breaks YARN , what am I doing wrong here?

yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.queues=Engineering,Marketing,Support
yarn.scheduler.capacity.root.Engineering.acl_administer_jobs=*
yarn.scheduler.capacity.root.Engineering.acl_submit_applications=*
yarn.scheduler.capacity.root.Engineering.capacity=100
yarn.scheduler.capacity.root.Engineering.maximum-capacity=100
yarn.scheduler.capacity.root.Engineering.state=RUNNING
yarn.scheduler.capacity.root.Engineering.user-limit-factor=1
yarn.scheduler.capacity.Engineering.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.Marketing.acl_administer_jobs=*
yarn.scheduler.capacity.root.Marketing.acl_submit_applications=*
yarn.scheduler.capacity.root.Marketing.capacity=100
yarn.scheduler.capacity.root.Marketing.maximum-capacity=100
yarn.scheduler.capacity.root.Marketing.state=RUNNING
yarn.scheduler.capacity.root.Marketing.user-limit-factor=1
yarn.scheduler.capacity.Marketing.minimum-user-limit-percent=100

[#2]

Hello Sami

Without the logs and or error message this is left up to guess. A quick look seems to show that your are allocating more than the total amount of ressources and that you do not seem to have defined the support queue you listed.

[#3 · Accepted solution]

@Sami Ahmad

I tried your config and the ResourceManager's logs show error: "Illegal capacity of -1.0 for queue root.Support".

Support is defined in root.queues, but it's capacity is undefined. If you fixed that (add yarn.scheduler.capacity.root.Support.capacity) the next problem will be, that the sum of the capacities of any queue's child queues must be 100%. In your case, root.Marketing + root.Engineering = 200%. Try the following:

yarn.scheduler.capacity.root.Engineering.capacity=40
yarn.scheduler.capacity.root.Marketing.capacity=40
yarn.scheduler.capacity.root.Support.capacity=20

or any other 3 numbers that add up to 100.

If you fix this too, your config will be valid: