from where we get the Yarn memory value from the dashboard

[#1]

we have Ambari cluster with 5 workers machines ( each worker have 32G memory )

in the dashboard we see the yarn memory like this:

77% used 575G of 746G

seems that the running application take 575G

but how 575G if total workers have 32X5=160G

and how the calculte is 746G ? on total ?

[#2 · Accepted solution]

@uri ben-ari,

746.5Gb is the total memory of all node managers included.

It is calculated as (yarn.nodemanager.resource.memory-mb) * (no of node managers) . Please check the value for yarn.nodemanager.resource.memory-mb in your YARN configs. Check the screenshot

575 GB is the memory used by all the applications. You can check the Resource manager UI to see which apps are using memory.

In the screenshot, you can see 3 apps running with one app taking 9Gb, and two apps taking 1Gb each. Therefore the total is 11Gb. You can check the similar in your environment.

Hope this helps.

Thanks,

Aditya

[#3]

I am little confuse in the value from yarn.nodemanager.resource.memory-mb is more then 120G and each worker machine only 32G , so how it can be that yarn.nodemanager.resource.memory-mb value much more the memory size of the worker machine ?

[#4]

@uri ben-ari,

This value is calculated using the stack advisor

'yarn.nodemanager.resource.memory-mb' = int(round(min(clusterData['containers']* clusterData['ramPerContainer'], nodemanagerMinRam))))

[#5]

I am little confuse in the value from yarn.nodemanager.resource.memory-mb is more then 120G and each worker machine only 32G , so how it can be that yarn.nodemanager.resource.memory-mb value much more the memory size of the worker machine ?