Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Migrating from Fair Scheduler to Capacity Scheduler

avatar
Master Guru

@Tim Hall

Any best practices when Migrating from Fair Scheduler to Capacity Scheduler?

1 ACCEPTED SOLUTION

avatar
Master Mentor

this is best practices for ordering algorithms and not migration per se, you may find more information by reading the Capacity Scheduler document http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...

10.3. Best Practices for Ordering Policies

  • Ordering policies are configured on a per-queue basis, with the default ordering policy set to FIFO. Fairness is usually best for on-demand, interactive, or exploratory workloads, while FIFO can be more efficient for predictable, recurring batch processing. You should segregate these different types of workloads into queues configured with the appropriate ordering policy.
  • In queues supporting both large and small applications, large applications can potentially "starve" (not receive sufficient resources). To avoid this scenario, use different queues for large and small jobs, or use size-based weighting to reduce the natural tendency of the ordering logic to favor smaller applications.
  • Use the yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent property to restrict the number of concurrent applications running in the queue to avoid a scenario in which too many applications are running simultaneously. Limits on each queue are directly proportional to their queue capacities and user limits. This property is specified as a float, for example: 0.5 = 50%. The default setting is 10%. This property can be set for all queues using theyarn.scheduler.capacity.maximum-am-resource-percent property, and can also be overridden on a per-queue basis using the yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent property.

View solution in original post

2 REPLIES 2

avatar
Master Mentor

this is best practices for ordering algorithms and not migration per se, you may find more information by reading the Capacity Scheduler document http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...

10.3. Best Practices for Ordering Policies

  • Ordering policies are configured on a per-queue basis, with the default ordering policy set to FIFO. Fairness is usually best for on-demand, interactive, or exploratory workloads, while FIFO can be more efficient for predictable, recurring batch processing. You should segregate these different types of workloads into queues configured with the appropriate ordering policy.
  • In queues supporting both large and small applications, large applications can potentially "starve" (not receive sufficient resources). To avoid this scenario, use different queues for large and small jobs, or use size-based weighting to reduce the natural tendency of the ordering logic to favor smaller applications.
  • Use the yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent property to restrict the number of concurrent applications running in the queue to avoid a scenario in which too many applications are running simultaneously. Limits on each queue are directly proportional to their queue capacities and user limits. This property is specified as a float, for example: 0.5 = 50%. The default setting is 10%. This property can be set for all queues using theyarn.scheduler.capacity.maximum-am-resource-percent property, and can also be overridden on a per-queue basis using the yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent property.

avatar
Master Mentor

furthermore, there's a great how-to guide for setting up capacity scheduler https://community.hortonworks.com/articles/238/configuring-yarn-capacity-scheduler-with-ambari.html