question Resource quotas within a single YARN queue in Support Questions

Resource quotas within a single YARN queue

fernando_lopez — Sat, 12 Jan 2019 04:37:55 GMT

We have defined several YARN queues. Say that you have queue Q1, where users A and B run Spark processes.

If A submits a job that demands all of the queue resources, they are allocated by YARN. Subsequently, when B submits his job, he is affected by resource scarcity.

We need to prevent this situation, by assigning resources more evenly between A and B (and all other incoming users), within Q1. We have already set Scheduler to Fair. Can this eager resource allocation behaviour be prevented?

Re: Resource quotas within a single YARN queue

Shelton — Sat, 12 Jan 2019 06:56:15 GMT

@Fernando Lopez Bello

Can you share your CapacityScheduler config

Re: Resource quotas within a single YARN queue

fernando_lopez — Mon, 14 Jan 2019 20:35:01 GMT

yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=10
yarn.scheduler.capacity.root.default.maximum-capacity=30
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=2
yarn.scheduler.capacity.root.queues=Hive,Zeppelin,default
yarn.scheduler.capacity.queue-mappings=u:zeppelin:Zeppelin,u:hdfs:Hive,g:dl-analytics-group:Zeppelin
yarn.scheduler.capacity.queue-mappings-override.enable=false
yarn.scheduler.capacity.root.Hive.acl_administer_queue=*
yarn.scheduler.capacity.root.Hive.acl_submit_applications=*
yarn.scheduler.capacity.root.Hive.capacity=50
yarn.scheduler.capacity.root.Hive.maximum-capacity=90
yarn.scheduler.capacity.root.Hive.minimum-user-limit-percent=25
yarn.scheduler.capacity.root.Hive.ordering-policy=fair
yarn.scheduler.capacity.root.Hive.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.Hive.priority=10
yarn.scheduler.capacity.root.Hive.state=RUNNING
yarn.scheduler.capacity.root.Hive.user-limit-factor=2
yarn.scheduler.capacity.root.Zeppelin.acl_administer_queue=*
yarn.scheduler.capacity.root.Zeppelin.acl_submit_applications=*
yarn.scheduler.capacity.root.Zeppelin.capacity=40
yarn.scheduler.capacity.root.Zeppelin.maximum-capacity=80
yarn.scheduler.capacity.root.Zeppelin.minimum-user-limit-percent=20
yarn.scheduler.capacity.root.Zeppelin.ordering-policy=fair
yarn.scheduler.capacity.root.Zeppelin.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.Zeppelin.priority=5
yarn.scheduler.capacity.root.Zeppelin.state=RUNNING
yarn.scheduler.capacity.root.Zeppelin.user-limit-factor=3
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=25
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.priority=0
yarn.scheduler.capacity.root.maximum-capacity=100
yarn.scheduler.capacity.root.ordering-policy=priority-utilization
yarn.scheduler.capacity.root.priority=0

Re: Resource quotas within a single YARN queue

npallapotu — Tue, 15 Jan 2019 04:37:28 GMT

@Fernando Lopez Bello Based on the above configuration file for queue Zeppelin although user A submits a job first utilizes all the resources as the Queue's minimum-user-limit-percent is set to 20 the queue resources will be shared among subsequent users below is the link which explains with an example.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_yarn-resource-management/content/setting_user_limits.html

But if you don't want all the resources to shared by user A even if there no other users then you can use user-limit-factor below is the link to a nice article about it. I can see user limit factor is 3 for Zeppelin Queue which means each user can utilize 3 times of queue capacity if resources are available & elasticity permits.

https://community.hortonworks.com/content/supportkb/49640/what-does-the-user-limit-factor-do-when-used-in-ya.html

In a nutshell minimum-user-limit-percent is a soft limit & user-limit-factor is a hard limit.