- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Configure preemption in Yarn Fair Scheduler
- Labels:
-
Apache Hadoop
-
Apache YARN
Created 09-13-2016 03:45 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All, Is there any reference doc for configuring pre-emption is YARN Fair Scheduler?
Created 09-13-2016 04:03 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Arun, FairScheduler is not HDP recommended/supported resource scheduler, so we don't have document to cover it. Please refer apache one: https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/FairScheduler.html.
I would like to add more background here: preemption feature is firstly added into CapacityScheduler which is quit mature and production ready. For FairScheduler, I am not exactly sure status (alpha or GA) but just notice several fixes are going on in community: https://issues.apache.org/jira/browse/YARN-4752
Any special reason to use FairScheduler? If not, you can also try preemption for CapacityScheduler. Here is doc: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_yarn_resource_mgt/content/preemption.htm...
Created 09-13-2016 03:51 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
HDP dont support Fair Scheduler, it support only capacity scheduler, you can check the following documents to configure pre-emption in capacity scheduler
http://hortonworks.com/blog/better-slas-via-resource-preemption-in-yarns-capacityscheduler/
Created 09-13-2016 03:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Rajkumar Singh, does that mean that setting the class to org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler does not take into effect?
Created 09-13-2016 04:21 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Arun A K it does take effect after setting yarn.resourcemanager.scheduler.class value to org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler using ambari but there is no property in ambari with the name of yarn.scheduler.fair.allocation.file so you need to define the custom property with the name of yarn.scheduler.fair.allocation.file and set it to fair-scheduler.xml.
Created 09-13-2016 04:24 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Rajkumar Singh, okay got it. One more related question - On which node should I be placing the fair allocation xml file?
Created 09-13-2016 04:25 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
on RM node and restart the RM.
Created 09-13-2016 04:03 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Arun, FairScheduler is not HDP recommended/supported resource scheduler, so we don't have document to cover it. Please refer apache one: https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/FairScheduler.html.
I would like to add more background here: preemption feature is firstly added into CapacityScheduler which is quit mature and production ready. For FairScheduler, I am not exactly sure status (alpha or GA) but just notice several fixes are going on in community: https://issues.apache.org/jira/browse/YARN-4752
Any special reason to use FairScheduler? If not, you can also try preemption for CapacityScheduler. Here is doc: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_yarn_resource_mgt/content/preemption.htm...
Created 09-13-2016 04:20 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks @Junping Du
Created 09-13-2016 04:18 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Arun - This post describes how you can use preemption with a fair policy in the capacity scheduler. It should give you a similar behavior to the fair scheduler but using a YARN capacity queue.
Created 09-13-2016 04:21 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks Chris. I will have a look.