I need to manage three queues in YARN: Production (PROD), Development (DEV) and Researching (LABS).
PROD process will require 50% of the cluster resources only two days per month. And DEV and LABS requieres the 50% of resource for each during the entire month. I want to have running DEV/LABS process with 50% for each one and for two days redistribute the consumption to have PROD: 50%, DEV: 25%, and LABS: 25%.
Do you have an idea how to have it? Thanks in advance.
@Facundo Bianco For the PROD queue you can specify yarn.scheduler.capacity.<prod_queue_path>.capacity to be 50%. For DEV and LABS each you can specify yarn.scheduler.capacity.<dev/labs_queue_path>.capacity to be 25% and yarn.scheduler.capacity.<dev/labs_queue_path>.maximum-capacity as 50% each. This will provide the required elasticity, i.e. when PROD is not being used the other two can use up to 50% each.
@Facundo Bianco you could define,
1. Parent queue PRP(pre production) min:50%, max:100% which has two leaf queue DEV and LABS. Both leaf queue will have min:50% and max:100%
2. PRD(production) with min:50%, max:50%.
In this set up, child queue for example DEV could use 100% resouces if its idle.
But for this setup we need to have pre-emption enable.