Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Configure preemption in Yarn Fair Scheduler

avatar
Super Collaborator

Hi All, Is there any reference doc for configuring pre-emption is YARN Fair Scheduler?

1 ACCEPTED SOLUTION

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
9 REPLIES 9

avatar
Super Guru

HDP dont support Fair Scheduler, it support only capacity scheduler, you can check the following documents to configure pre-emption in capacity scheduler

http://hortonworks.com/blog/better-slas-via-resource-preemption-in-yarns-capacityscheduler/

avatar
Super Collaborator

@Rajkumar Singh, does that mean that setting the class to org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler does not take into effect?

avatar
Super Guru

@Arun A K it does take effect after setting yarn.resourcemanager.scheduler.class value to org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler using ambari but there is no property in ambari with the name of yarn.scheduler.fair.allocation.file so you need to define the custom property with the name of yarn.scheduler.fair.allocation.file and set it to fair-scheduler.xml.

avatar
Super Collaborator

@Rajkumar Singh, okay got it. One more related question - On which node should I be placing the fair allocation xml file?

avatar
Super Guru

on RM node and restart the RM.

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Super Collaborator

Thanks @Junping Du

avatar

Hi Arun - This post describes how you can use preemption with a fair policy in the capacity scheduler. It should give you a similar behavior to the fair scheduler but using a YARN capacity queue.

https://community.hortonworks.com/articles/44079/yarn-pre-emption-with-spark-using-a-fair-policy.htm...

avatar
Super Collaborator

Thanks Chris. I will have a look.