Support Questions

Find answers, ask questions, and share your expertise

[YARN] Fair scheduler priority


I need to manage three queues in YARN: Production (PROD), Development (DEV) and Researching (LABS).

PROD process will require 50% of the cluster resources only two days per month. And DEV and LABS requieres the 50% of resource for each during the entire month. I want to have running DEV/LABS process with 50% for each one and for two days redistribute the consumption to have PROD: 50%, DEV: 25%, and LABS: 25%.


Do you have an idea how to have it? Thanks in advance.



@Facundo Bianco For the PROD queue you can specify yarn.scheduler.capacity.<prod_queue_path>.capacity to be 50%. For DEV and LABS each you can specify yarn.scheduler.capacity.<dev/labs_queue_path>.capacity to be 25% and yarn.scheduler.capacity.<dev/labs_queue_path>.maximum-capacity as 50% each. This will provide the required elasticity, i.e. when PROD is not being used the other two can use up to 50% each.


@Facundo Bianco you could define,

1. Parent queue PRP(pre production) min:50%, max:100% which has two leaf queue DEV and LABS. Both leaf queue will have min:50% and max:100%

2. PRD(production) with min:50%, max:50%.

In this set up, child queue for example DEV could use 100% resouces if its idle.

But for this setup we need to have pre-emption enable.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.