Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

[YARN] Fair scheduler priority

Highlighted

[YARN] Fair scheduler priority

Contributor

I need to manage three queues in YARN: Production (PROD), Development (DEV) and Researching (LABS).

PROD process will require 50% of the cluster resources only two days per month. And DEV and LABS requieres the 50% of resource for each during the entire month. I want to have running DEV/LABS process with 50% for each one and for two days redistribute the consumption to have PROD: 50%, DEV: 25%, and LABS: 25%.

16583-yarn-queues-1.png

Do you have an idea how to have it? Thanks in advance.

2 REPLIES 2

Re: [YARN] Fair scheduler priority

Contributor

@Facundo Bianco For the PROD queue you can specify yarn.scheduler.capacity.<prod_queue_path>.capacity to be 50%. For DEV and LABS each you can specify yarn.scheduler.capacity.<dev/labs_queue_path>.capacity to be 25% and yarn.scheduler.capacity.<dev/labs_queue_path>.maximum-capacity as 50% each. This will provide the required elasticity, i.e. when PROD is not being used the other two can use up to 50% each.

Re: [YARN] Fair scheduler priority

Contributor

@Facundo Bianco you could define,

1. Parent queue PRP(pre production) min:50%, max:100% which has two leaf queue DEV and LABS. Both leaf queue will have min:50% and max:100%

2. PRD(production) with min:50%, max:50%.

In this set up, child queue for example DEV could use 100% resouces if its idle.

But for this setup we need to have pre-emption enable.

Don't have an account?
Coming from Hortonworks? Activate your account here