I need to manage three queues in YARN: Production (PROD), Development (DEV) and Researching (LABS).
PROD process will require 50% of the cluster resources only two days per month. And DEV and LABS requieres the 50% of resource for each during the entire month. I want to have running DEV/LABS process with 50% for each one and for two days redistribute the consumption to have PROD: 50%, DEV: 25%, and LABS: 25%.
Do you have an idea how to have it? Thanks in advance.
@Facundo Bianco For the PROD queue you can specify yarn.scheduler.capacity.<prod_queue_path>.capacity to be 50%. For DEV and LABS each you can specify yarn.scheduler.capacity.<dev/labs_queue_path>.capacity to be 25% and yarn.scheduler.capacity.<dev/labs_queue_path>.maximum-capacity as 50% each. This will provide the required elasticity, i.e. when PROD is not being used the other two can use up to 50% each.