Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Determine memory settings for YARN

Determine memory settings for YARN

New Contributor



we are currently setting up a cluster of 10 machines, each equipped with 24 cores Intel Xeon 3.4 Ghz, 128 GB of main memory and 8x3TB local disks.


I am a bit confused about the memory settings for YARN that are best in this setting. What we have is



Map Task Memory ( = 5555 MiB

Map Task CPU Virtual Cores ( = 1

Reduce Task Memory (mapreduce.reduce.memory.mb) = 8 Gib

Reduce Task CPU Virtual Cores (mapreduce.reduce.cpu.vcores) = 1

Map Task Maximum Heap Size ( = 4444 MiB

Reduce Task Maximum Heap Size ( = 6553 MiB

Client Java Heap Size = 4 GiB

Container Memory (yarn.nodemanager.resource.memory-mb) = 64 GiB

Container Virtual CPU Cores = 8

Container Memory Minimum (yarn.scheduler.minimum-allocation-mb) =  2 GiB

Container Memory Maximum (yarn.scheduler.maximum-allocation-mb) = 64 GiB


On the cluster, we are running several services (HBase, Hive, Impala, Spark).


With these settings in place, we are getting the following warning:


Memory on host is overcommitted. The total memory allocation is 156.4 GiB bytes but there are only 125.9 GiB bytes of RAM (25.2 GiB bytes of which are reserved for the system). Visit the Resources tab on the Host page for allocation details. Reconfigure the roles on the host to lower the overall memory allocation. Note: Java maximum heap sizes are multiplied by 1.3 to approximate JVM overhead.



This seems to come specifically from the setting 


Container Memory (yarn.nodemanager.resource.memory-mb) = 64 GiB


Actually, we get rid of the warning when we set 


Container Memory (yarn.nodemanager.resource.memory-mb) = 8 GiB


However, from the setup described above, I had the impression that 64 GiB is the appropriate setting for the specified hardware. Can anyone comment on this?

Do the settings above make sense? 


Thank you very much in advance for your help and suggestions.


Kind regards,








Re: Determine memory settings for YARN

Master Guru
What other roles run on What are their heap sizes/memory limits? The warning you see is computed by considering all roles running on a given host, rather than just YARN.