How to allocate YARN memory to different users with priority

[#1]

Hi @Jay Kumar SenSharma

I have 5 node(HD 4TB, RAM 64GB) HDP cluster. Around 5 different users access the cluster.

For example: user1, user2, user3,user4,user5.

I want to highest priority to user1. Whenever a query is fired from user1 it should be executed first.

Currently what ever user's query comes first YARN memory is 100% and fully utilized by the user.

Reaming users task is staying in QUEUE. once the task is finished and YARN memory is released till that time all the querys are waiting in QUEUE.

How can i allocate the YARN memory to each user and give priority to each user??

[#2]

@kotesh banoth, you can setup user-queue mapping in capacity scheduler. This will restrict access to the queue as per users. Once this mapping is set, you can set priority to respective queue. As per above mentioned case, the queue related to user1 should have highest priority, this way application started in this queue will execute first.

user-queue mapping:

http://tamastarjanyi.blogspot.com/2015/01/user-based-queue-mapping-for-capacity.html

Queue priority setting:

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-views/content/setting_yarn_queue_...

[#3]

You try following

1) Enable the Scheduler in YARN with Capacity Scheduler.

https://hortonworks.com/hadoop-tutorial/configuring-yarn-capacity-scheduler-ambari/

2) Allocate the resource on respective user level , consider we having two user. allocate as below.

###Default user #####
yarn.scheduler.capacity.root.default.capacity=10
yarn.scheduler.capacity.root.default.maximum-applications=10000
yarn.scheduler.capacity.root.default.maximum-capacity=30
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=1
## user1 #####
yarn.scheduler.capacity.root.user1.acl_submit_applications=*
yarn.scheduler.capacity.root.user1.capacity=21
yarn.scheduler.capacity.root.user1.maximum-capacity=35
yarn.scheduler.capacity.root.user1.ordering-policy=fair
yarn.scheduler.capacity.root.user1.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.user1.state=RUNNING
yarn.scheduler.capacity.root.user1.user-limit-factor=1
yarn.scheduler.capacity.root.user2.acl_submit_applications=*
## user2 #####
yarn.scheduler.capacity.root.user2.capacity=20
yarn.scheduler.capacity.root.user2.maximum-capacity=35
yarn.scheduler.capacity.root.user2.ordering-policy=fair
yarn.scheduler.capacity.root.user2.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.user2.state=RUNNING
yarn.scheduler.capacity.root.user2.user-limit-factor=1

3) Restart the YARN and MAPREDUCE services.

4) Verify in RM capacity is allocated , in Schedular TAB

[#4]

Hi @zkfs, @yvora @Jay Kumar SenSharma

I had made the changes as below(attached the text file) in YARN scheduler.

yarn-resource-allocation.txt

After making the changes and restarting the YARN the ResourceManager is going down within couple of minutes after restarting YARN.

Kindly help me out. as single query is taking full YARN memory.

Thanks in advance.

[#5]

Please share error log information form the YARN going down ?

[#6]

Please share error information from the Yarn Log error

[#7]

@zkfs , @Jay Kumar SenSharma

My resource manager is going down after making the changes in the YARN Scheduler.

yarn-yarn-resourcemanager.log:

java.lang.IllegalStateException: Queue configuration missing child queue names for root
java.lang.IllegalStateException: Queue configuration missing child queue names for root
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:591)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:468)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:271)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:302)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:592)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1006)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:266)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1236)
2017-12-28 15:06:41,788 INFO  resourcemanager.ResourceManager (ResourceManager.java:transitionToStandby(1078)) - Transitioning to standby state
2017-12-28 15:06:41,788 INFO  resourcemanager.ResourceManager (ResourceManager.java:transitionToStandby(1085)) - Transitioned to standby state
2017-12-28 15:06:41,788 FATAL resourcemanager.ResourceManager (ResourceManager.java:main(1240)) - Error starting ResourceManager
java.lang.IllegalStateException: Queue configuration missing child queue names for root
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:591)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:468)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:271)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:302)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:592)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1006)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:266)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1236)
2017-12-28 15:06:41,790 INFO  resourcemanager.ResourceManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at chtcuxhd03/172.16.0.123
************************************************************/

I have attached full yarn resource configuration and yarn resource manager log

scheduler-configuration.txt yarn-yarn-resourcemanager-chtcuxhd03.txt

[#8]

Please folow steps as below and verify is required parameter as inplace or not .

https://hortonworks.com/hadoop-tutorial/configuring-yarn-capacity-scheduler-ambari/

Correction:- Once you updated the YARN Queue, required to restart the all service.

Note: Kindly do test on Non-prod or personal computer