problem with yarn from ambari dashboard

[#1]

what could be the reason that yarn memory is very high?

any suggestion to verify this?

we have this value from ambari

yarn.scheduler.capacity.root.default.user-limit-factor=1

yarn.scheduler.minimum-allocation-mb=11776

yarn.scheduler.maximum-allocation-mb=122880

yarn.nodemanager.resource.memory-mb=120G

 /usr/bin/yarn application -list -appStates RUNNING | grep  RUNNING


  Thrift JDBC/ODBC Server                SPARK          hive         default                 RUNNING               UNDEFINED                  10%              
  Thrift JDBC/ODBC Server                SPARK          hive         default                 RUNNING               UNDEFINED                  10%           
  mcMFM                                  SPARK          hdfs         default                 RUNNING               UNDEFINED                  10%             
  mcMassRepo                             SPARK          hdfs         default                 RUNNING               UNDEFINED                  10%           
  mcMassProfiling                        SPARK          hdfs         default                 RUNNING               UNDEFINED




free -g
              total        used        free      shared  buff/cache   available
Mem:             31          24           0           1           6           5
Swap:             7           0           7

[#2 · Accepted solution]

@uri ben-ari

If you have not set it on your own then , I guess based on the Stack Advisor script as:

https://github.com/apache/ambari/blob/release-2.5.2/ambari-server/src/main/resources/stacks/HDP/2.0....

    putYarnProperty('yarn.nodemanager.resource.memory-mb', int(round(min(clusterData['containers'] * clusterData['ramPerContainer'], nodemanagerMinRam))))

.

[#3]

do you mean that I need to change the yarn.nodemanager.resource.memory-m to other value? if yes how to calculate this value ? ( or maybe we need to increase this value for example to 200G )

[#4]

another question once I change this value - yarn.nodemanager.resource.memory-mbthe , then yarn memory value is now 100% will immediately decrease ? or need to do some other action to refresh it ?

[#5]

@uri ben-ari

I will have to check with some Yarn experts. I will check and update.

[#6]

@uri ben-ari

This is the response i got from @pjoseph

Example:

- The "yarn.nodemanager.resource.memory-mb" is NodeManager memory reserved for yarn tasks.
- So for example, the machine RAM is 300GB and you reserve 50GB for OS, Dameons and provide 250GB to yarn
- So i configure "yarn.nodemanager.resource.memory-mb" to 250GB
- Say i have 5 nodes, each node will register to RM once at startup
- So RM will have total memory 250 * 5 = 1250GB

- In case m increasing the yarn.nodemanager.resource.memory-mb value to 260GB and restarting NM now, 260 * 5 will be total

.

[#7]

@uri ben-ari

You can also have a look at the "The HDP utility script is the recommended method for calculating HDP memory configuration settings"

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_command-line-installation/content/determ...

.

[#8]

ok thank you waiting for your answer

[#9]

I update the question with more info ( like which application are runing under yarn and machine memory )

[#10]

so on each worker machine we have 32G ( lets say we allocate 7 to the OS ) so we have 25G , in my system we have 5 workers so this mean 25*5=125 , and this is actually what configured , am I corect ? ( we have in the cluster 3 master machines , 5 workers machines ) , in that case we need to increase each worker memory to at least 50G - am I correct ? and after that set the value of yarn.nodemanager.resource.memory-mb to 260G for example