- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Migrating from Fair Scheduler to Capacity Scheduler
- Labels:
-
Apache YARN
Created ‎01-05-2016 04:43 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Any best practices when Migrating from Fair Scheduler to Capacity Scheduler?
Created ‎01-05-2016 02:06 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
this is best practices for ordering algorithms and not migration per se, you may find more information by reading the Capacity Scheduler document http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...
10.3. Best Practices for Ordering Policies
- Ordering policies are configured on a per-queue basis, with the default ordering policy set to FIFO. Fairness is usually best for on-demand, interactive, or exploratory workloads, while FIFO can be more efficient for predictable, recurring batch processing. You should segregate these different types of workloads into queues configured with the appropriate ordering policy.
- In queues supporting both large and small applications, large applications can potentially "starve" (not receive sufficient resources). To avoid this scenario, use different queues for large and small jobs, or use size-based weighting to reduce the natural tendency of the ordering logic to favor smaller applications.
- Use the
yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent
property to restrict the number of concurrent applications running in the queue to avoid a scenario in which too many applications are running simultaneously. Limits on each queue are directly proportional to their queue capacities and user limits. This property is specified as a float, for example: 0.5 = 50%. The default setting is 10%. This property can be set for all queues using theyarn.scheduler.capacity.maximum-am-resource-percent
property, and can also be overridden on a per-queue basis using theyarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent
property.
Created ‎01-05-2016 02:06 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
this is best practices for ordering algorithms and not migration per se, you may find more information by reading the Capacity Scheduler document http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...
10.3. Best Practices for Ordering Policies
- Ordering policies are configured on a per-queue basis, with the default ordering policy set to FIFO. Fairness is usually best for on-demand, interactive, or exploratory workloads, while FIFO can be more efficient for predictable, recurring batch processing. You should segregate these different types of workloads into queues configured with the appropriate ordering policy.
- In queues supporting both large and small applications, large applications can potentially "starve" (not receive sufficient resources). To avoid this scenario, use different queues for large and small jobs, or use size-based weighting to reduce the natural tendency of the ordering logic to favor smaller applications.
- Use the
yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent
property to restrict the number of concurrent applications running in the queue to avoid a scenario in which too many applications are running simultaneously. Limits on each queue are directly proportional to their queue capacities and user limits. This property is specified as a float, for example: 0.5 = 50%. The default setting is 10%. This property can be set for all queues using theyarn.scheduler.capacity.maximum-am-resource-percent
property, and can also be overridden on a per-queue basis using theyarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent
property.
Created ‎01-05-2016 02:10 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
furthermore, there's a great how-to guide for setting up capacity scheduler https://community.hortonworks.com/articles/238/configuring-yarn-capacity-scheduler-with-ambari.html
