question Re: YARN Resource allocation for containers using yarn.scheduler.minimum-mb in Archives of Support Questions (Read Only)

YARN Resource allocation for containers using yarn.scheduler.minimum-mb

sivasankar_chan — Thu, 16 Aug 2018 23:07:33 GMT

Hi,

I have the doubt regarding the below properties could you please clarify this for my clear and better understanding.

yarn.scheduler.minimum-allocation-mb

yarn.scheduler.maximum-allocation-mb

These 2 are the properties which will decide the minimum and maximum memory allocation per container. But my doubt here is i might have data nodes with different memory (means 8 GB/16GB/32 GB) in the same cluster. Then how do i set this yarn.scheduler.maximum.allocation-mb. Do i need to set this for node in the cluster. how this will work. Please help me to understand better.

Thanx

Siva

Re: YARN Resource allocation for containers using yarn.scheduler.minimum-mb

apappu — Thu, 16 Aug 2018 23:11:12 GMT

@Sivasankar Chandrasekar may be you can create config groups in Ambari and then define it.

Re: YARN Resource allocation for containers using yarn.scheduler.minimum-mb

TarunParimi — Thu, 16 Aug 2018 23:51:17 GMT

@Sivasankar Chandrasekar

yarn.scheduler.maximum.allocation-mb is a scheduler level config and applies to the ResourceManager only. It should be set to a single value, ideally the largest possible container your applications may want to request. You can set it to 8GB if your applications will only use the maximum of 8GB for a single container. If you have requirement of launching a single container of size 32GB, you can also set it to 32GB. But only nodes which have 32GB memory can fulfill the container request.

You should also create config groups in ambari for the property yarn.nodemanager.resource.memory-mb and set to different values 8 GB/16GB/32 GB with respect to the nodes.

Re: YARN Resource allocation for containers using yarn.scheduler.minimum-mb

sivasankar_chan — Fri, 17 Aug 2018 14:45:06 GMT

Thank you Tarun Parimi.