Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

YARN Resource allocation for containers using yarn.scheduler.minimum-mb

avatar

Hi,

I have the doubt regarding the below properties could you please clarify this for my clear and better understanding.

yarn.scheduler.minimum-allocation-mb
yarn.scheduler.maximum-allocation-mb

These 2 are the properties which will decide the minimum and maximum memory allocation per container. But my doubt here is i might have data nodes with different memory (means 8 GB/16GB/32 GB) in the same cluster. Then how do i set this yarn.scheduler.maximum.allocation-mb. Do i need to set this for node in the cluster. how this will work. Please help me to understand better.

Thanx

Siva

1 ACCEPTED SOLUTION

avatar
Expert Contributor
@Sivasankar Chandrasekar

yarn.scheduler.maximum.allocation-mb is a scheduler level config and applies to the ResourceManager only. It should be set to a single value, ideally the largest possible container your applications may want to request. You can set it to 8GB if your applications will only use the maximum of 8GB for a single container. If you have requirement of launching a single container of size 32GB, you can also set it to 32GB. But only nodes which have 32GB memory can fulfill the container request.

You should also create config groups in ambari for the property yarn.nodemanager.resource.memory-mb and set to different values 8 GB/16GB/32 GB with respect to the nodes.

View solution in original post

3 REPLIES 3

avatar

@Sivasankar Chandrasekar may be you can create config groups in Ambari and then define it.

avatar
Expert Contributor
@Sivasankar Chandrasekar

yarn.scheduler.maximum.allocation-mb is a scheduler level config and applies to the ResourceManager only. It should be set to a single value, ideally the largest possible container your applications may want to request. You can set it to 8GB if your applications will only use the maximum of 8GB for a single container. If you have requirement of launching a single container of size 32GB, you can also set it to 32GB. But only nodes which have 32GB memory can fulfill the container request.

You should also create config groups in ambari for the property yarn.nodemanager.resource.memory-mb and set to different values 8 GB/16GB/32 GB with respect to the nodes.

avatar

Thank you Tarun Parimi.