Migrating from Fair Scheduler to Capacity Scheduler

sunile_manjee — Tue, 05 Jan 2016 12:43:08 GMT

Any best practices when Migrating from Fair Scheduler to Capacity Scheduler?

Re: Migrating from Fair Scheduler to Capacity Scheduler

aervits — Tue, 05 Jan 2016 22:06:01 GMT

this is best practices for ordering algorithms and not migration per se, you may find more information by reading the Capacity Scheduler document http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...

Ordering policies are configured on a per-queue basis, with the default ordering policy set to FIFO. Fairness is usually best for on-demand, interactive, or exploratory workloads, while FIFO can be more efficient for predictable, recurring batch processing. You should segregate these different types of workloads into queues configured with the appropriate ordering policy.
In queues supporting both large and small applications, large applications can potentially "starve" (not receive sufficient resources). To avoid this scenario, use different queues for large and small jobs, or use size-based weighting to reduce the natural tendency of the ordering logic to favor smaller applications.
Use the yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent property to restrict the number of concurrent applications running in the queue to avoid a scenario in which too many applications are running simultaneously. Limits on each queue are directly proportional to their queue capacities and user limits. This property is specified as a float, for example: 0.5 = 50%. The default setting is 10%. This property can be set for all queues using theyarn.scheduler.capacity.maximum-am-resource-percent property, and can also be overridden on a per-queue basis using the yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent property.

aervits — Tue, 05 Jan 2016 22:10:04 GMT