NodeManager memory setting best practice?

[#1]

I asked a question regarding OOM in NodeManager log and it turned out I need to increase the NM's heap size.

So that I was wondering if there is any document which advises me how much memory I should assign?

I'm hoping something similar to HDFS NameNode heap size ( http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-... )

Or as it would be hard to estimate, I should just try twice higher and see how it goes?

[#2]

Hi Hajime, typically we set the NodeManager heap to 2GB - 4GB - I haven't had to set it higher than that. What is it currently set to?

-Koelli

[#3]

2GB (i wonder why this system has min char limit which doesn't allow me to just type "2GB")

[#4]

@Hajime - The best way to find the nodemanager heap size and other memory settings is to calculate it specifically for your cluster size and hardware spec. Here is the utility that you can use -

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...

Snippet 

hdp-configuration-utils.sh options where options are as follows:

Table 1.1. hdp-configuration-utils.sh Options

Option

Description

-c CORESThe number of cores on each host.-m MEMORYThe amount of memory on each host in GB.-d DISKSThe number of disks on each host.-k HBASE"True" if HBase is installed, "False" if not.

The output recommendation is in this format -

Using cores=16 memory=64GB disks=4 hbase=True
Profile: cores=16 memory=49152MB reserved=16GB usableMem=48GB disks=4 
Num Container=8
Container Ram=6144MB 
Used Ram=48GB
Unused Ram=16GB
yarn.scheduler.minimum-allocation-mb=6144 
yarn.scheduler.maximum-allocation-mb=49152 
yarn.nodemanager.resource.memory-mb=49152 
mapreduce.map.memory.mb=6144 
mapreduce.map.java.opts=-Xmx4096m 
mapreduce.reduce.memory.mb=6144 
mapreduce.reduce.java.opts=-Xmx4096m 
yarn.app.mapreduce.am.resource.mb=6144 
yarn.app.mapreduce.am.command-opts=-Xmx4096m 
mapreduce.task.io.sort.mb=1792 
tez.am.resource.memory.mb=6144 
tez.am.launch.cmd-opts =-Xmx4096m 
hive.tez.container.size=6144 
hive.tez.java.opts=-Xmx4096m
hive.auto.convert.join.noconditionaltask.size=1342177000

[#5]

Thanks! but if i understand this correctly, it does not show NodeManager itself's heap (Xmx), does it?

[#6 · Accepted solution]

Hajime, the above scripts are for the yarn container and mapreduce memory settings. If you are trying to configure the memory of the nodemanager process itself then that shouldn't need more than 2GB - 4GB. If you are seeing outOfMemory there I suggest you turn on verbose GC for the nodemanager process and review the GC logs.