YARN Resource allocation for containers using yarn.scheduler.minimum-mb

[#1]

Hi,

I have the doubt regarding the below properties could you please clarify this for my clear and better understanding.

yarn.scheduler.minimum-allocation-mb

yarn.scheduler.maximum-allocation-mb

These 2 are the properties which will decide the minimum and maximum memory allocation per container. But my doubt here is i might have data nodes with different memory (means 8 GB/16GB/32 GB) in the same cluster. Then how do i set this yarn.scheduler.maximum.allocation-mb. Do i need to set this for node in the cluster. how this will work. Please help me to understand better.

Thanx

Siva

[#2]

@Sivasankar Chandrasekar may be you can create config groups in Ambari and then define it.

[#3 · Accepted solution]

@Sivasankar Chandrasekar

yarn.scheduler.maximum.allocation-mb is a scheduler level config and applies to the ResourceManager only. It should be set to a single value, ideally the largest possible container your applications may want to request. You can set it to 8GB if your applications will only use the maximum of 8GB for a single container. If you have requirement of launching a single container of size 32GB, you can also set it to 32GB. But only nodes which have 32GB memory can fulfill the container request.

You should also create config groups in ambari for the property yarn.nodemanager.resource.memory-mb and set to different values 8 GB/16GB/32 GB with respect to the nodes.

[#4]

Thank you Tarun Parimi.