Is there a way to set the yarn.nodemanager.resource.memory-mb and cpu-cores per node-label? I have 2 queues, and 2 node-labels. Each queue is attached with each node-label. One node-label contains a set of nodes with a lot of RAM and few cores, the other contains the opposite. How can I differentiate the NodeManagers deployed in these 2 queues? because right now, config groups aren’t making it work for me since the setting is in a resource-manager level. Are there any other ways?
Also is there a way where I can set the scheduler.class on a Queue Level, I know that it's ResourceManager level setting, just wanted to know if there's any hacks or ways to workaround this?
Yarn overall administers all the ressources of your hadoop cluster and config groups are the right way to specify differences like the ones you point to. By default all nodes belong the default node label. If you add other node labels to queue A for example lets say node label X and Y and create a rule such as capacity(a) = 40, capacity(a, label=x) = 100, capacity(a, label=y) = 50
What this means is you actually get on queue A 40% of the ressources with no label or label default and 100% of the ressources of machines with Label X and 50% of ressources of machines with Label Y
In your case your queue would only have the one label you defined and set to 100%. The memory and cpu ressources are then delegated to the Yarn properties and config groups you have set up.
maybe this doc helps illustrate the point: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_yarn_resource_mgt/content/configuring_no...
hope this helps
Thanks nmaillard, but yeah I understand that we can play around with the capacity value of each queue/node-label pair. But I was wondering if I can explicitly say: "node-label A" would have 16core/128gbRAM NodeManagers and "node-label B" would have 8core/32gbRAM NodeManagers?
Hello ace, you can't explicitly set these values since yarn's unit of management is containers. The only way to do this would be to express that granularity in the config group like max vcores 16 and max mem 128GB or have your node-label percentage reflect those figures in container ressources, say you have 100 vcores and 100 gb and you give 80% you are giving "80 vcores and 80 gb". This is a very simplistic break down but just for the argument's sake.