Support Questions
Find answers, ask questions, and share your expertise

from where we get the Yarn memory value from the dashboard

39970-capture.png

we have Ambari cluster with 5 workers machines ( each worker have 32G memory )

in the dashboard we see the yarn memory like this:

77% used 575G of 746G

seems that the running application take 575G

but how 575G if total workers have 32X5=160G

and how the calculte is 746G ? on total ?

Michael-Bronson
1 ACCEPTED SOLUTION

Accepted Solutions

@uri ben-ari,

746.5Gb is the total memory of all node managers included.

It is calculated as (yarn.nodemanager.resource.memory-mb) * (no of node managers) . Please check the value for yarn.nodemanager.resource.memory-mb in your YARN configs. Check the screenshot

39972-screen-shot-2017-10-26-at-101615-pm.png

575 GB is the memory used by all the applications. You can check the Resource manager UI to see which apps are using memory.

39973-screen-shot-2017-10-26-at-101859-pm.png

In the screenshot, you can see 3 apps running with one app taking 9Gb, and two apps taking 1Gb each. Therefore the total is 11Gb. You can check the similar in your environment.

Hope this helps.

Thanks,

Aditya

View solution in original post

4 REPLIES 4

@uri ben-ari,

746.5Gb is the total memory of all node managers included.

It is calculated as (yarn.nodemanager.resource.memory-mb) * (no of node managers) . Please check the value for yarn.nodemanager.resource.memory-mb in your YARN configs. Check the screenshot

39972-screen-shot-2017-10-26-at-101615-pm.png

575 GB is the memory used by all the applications. You can check the Resource manager UI to see which apps are using memory.

39973-screen-shot-2017-10-26-at-101859-pm.png

In the screenshot, you can see 3 apps running with one app taking 9Gb, and two apps taking 1Gb each. Therefore the total is 11Gb. You can check the similar in your environment.

Hope this helps.

Thanks,

Aditya

View solution in original post

I am little confuse in the value from yarn.nodemanager.resource.memory-mb is more then 120G and each worker machine only 32G , so how it can be that yarn.nodemanager.resource.memory-mb value much more the memory size of the worker machine ?

Michael-Bronson

@uri ben-ari,

This value is calculated using the stack advisor

'yarn.nodemanager.resource.memory-mb' = int(round(min(clusterData['containers']* clusterData['ramPerContainer'], nodemanagerMinRam))))

I am little confuse in the value from yarn.nodemanager.resource.memory-mb is more then 120G and each worker machine only 32G , so how it can be that yarn.nodemanager.resource.memory-mb value much more the memory size of the worker machine ?

Michael-Bronson