datanode with different RAM configurations...how to utilize the max memory

[#1]

Hi All,

I wanted to know what parameters needs to be changed we have cluster with nodes of different characteristics in RAM,

Eg: We have 143 datanodes with 256 GB RAM planning to upgrade few datanodes to 512 GB RAM, so how can i make sure YARN utilizes this change. Currently yarn is allocated 168 GB.

I researched and found this https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-operations/content/using_host_con... which says to add config groups

Is there any detailed document for this, as what settings needs changes.

[#2]

Hi @sudi ts

Create a "Config Group" under YARN Service.

Note, just increasing RAM capacity might not be enough. You may need additional CPUs to handle more jobs / containers.

yarn-site.xml
yarn.nodemanager.resource.memory-mb
yarn.nodemanager.resource.cpu-vcores

Add Node Manager Host to this config group

Restart Respective Nodemanagers associated to this config group.