Reply
New Contributor
Posts: 1
Registered: ‎07-20-2015

Determine memory settings for YARN

Hello,

 

we are currently setting up a cluster of 10 machines, each equipped with 24 cores Intel Xeon 3.4 Ghz, 128 GB of main memory and 8x3TB local disks.

 

I am a bit confused about the memory settings for YARN that are best in this setting. What we have is

 

 

Map Task Memory (mapreduce.map.memory.mb) = 5555 MiB

Map Task CPU Virtual Cores (mapreduce.map.cpu.vcores) = 1

Reduce Task Memory (mapreduce.reduce.memory.mb) = 8 Gib

Reduce Task CPU Virtual Cores (mapreduce.reduce.cpu.vcores) = 1

Map Task Maximum Heap Size (mapreduce.map.java.opts.max.heap) = 4444 MiB

Reduce Task Maximum Heap Size (mapreduce.reduce.java.opts.max.heap) = 6553 MiB

Client Java Heap Size = 4 GiB

Container Memory (yarn.nodemanager.resource.memory-mb) = 64 GiB

Container Virtual CPU Cores = 8

Container Memory Minimum (yarn.scheduler.minimum-allocation-mb) =  2 GiB

Container Memory Maximum (yarn.scheduler.maximum-allocation-mb) = 64 GiB

 

On the cluster, we are running several services (HBase, Hive, Impala, Spark).

 

With these settings in place, we are getting the following warning:

 

Memory on host d5hadoop22.mpi-inf.mpg.de is overcommitted. The total memory allocation is 156.4 GiB bytes but there are only 125.9 GiB bytes of RAM (25.2 GiB bytes of which are reserved for the system). Visit the Resources tab on the Host page for allocation details. Reconfigure the roles on the host to lower the overall memory allocation. Note: Java maximum heap sizes are multiplied by 1.3 to approximate JVM overhead.

 

 

This seems to come specifically from the setting 

 

Container Memory (yarn.nodemanager.resource.memory-mb) = 64 GiB

 

Actually, we get rid of the warning when we set 

 

Container Memory (yarn.nodemanager.resource.memory-mb) = 8 GiB

 

However, from the setup described above, I had the impression that 64 GiB is the appropriate setting for the specified hardware. Can anyone comment on this?

Do the settings above make sense? 

 

Thank you very much in advance for your help and suggestions.

 

Kind regards,

Stephan

 

 

 

 

 

Highlighted
Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: Determine memory settings for YARN

What other roles run on d5hadoop22.mpi-inf.mpg.de? What are their heap sizes/memory limits? The warning you see is computed by considering all roles running on a given host, rather than just YARN.
Announcements