question Re: problem with yarn from ambari dashboard in Archives of Support Questions (Read Only)

problem with yarn from ambari dashboard

mike_bronson7 — Sun, 18 Aug 2019 01:13:56 GMT

what could be the reason that yarn memory is very high?

any suggestion to verify this?

we have this value from ambari

yarn.scheduler.capacity.root.default.user-limit-factor=1

yarn.scheduler.minimum-allocation-mb=11776

yarn.scheduler.maximum-allocation-mb=122880

yarn.nodemanager.resource.memory-mb=120G


 /usr/bin/yarn application -list -appStates RUNNING | grep  RUNNING


  Thrift JDBC/ODBC Server                SPARK          hive         default                 RUNNING               UNDEFINED                  10%              
  Thrift JDBC/ODBC Server                SPARK          hive         default                 RUNNING               UNDEFINED                  10%           
  mcMFM                                  SPARK          hdfs         default                 RUNNING               UNDEFINED                  10%             
  mcMassRepo                             SPARK          hdfs         default                 RUNNING               UNDEFINED                  10%           
  mcMassProfiling                        SPARK          hdfs         default                 RUNNING               UNDEFINED




free -g
              total        used        free      shared  buff/cache   available
Mem:             31          24           0           1           6           5
Swap:             7           0           7

Re: problem with yarn from ambari dashboard

jsensharma — Wed, 25 Oct 2017 15:41:29 GMT

@uri ben-ari

If you have not set it on your own then , I guess based on the Stack Advisor script as:

https://github.com/apache/ambari/blob/release-2.5.2/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py#L130

    putYarnProperty('yarn.nodemanager.resource.memory-mb', int(round(min(clusterData['containers'] * clusterData['ramPerContainer'], nodemanagerMinRam))))

Re: problem with yarn from ambari dashboard

mike_bronson7 — Wed, 25 Oct 2017 15:48:37 GMT

do you mean that I need to change the yarn.nodemanager.resource.memory-m to other value? if yes how to calculate this value ? ( or maybe we need to increase this value for example to 200G )

Re: problem with yarn from ambari dashboard

mike_bronson7 — Wed, 25 Oct 2017 16:04:07 GMT

another question once I change this value - yarn.nodemanager.resource.memory-mbthe , then yarn memory value is now 100% will immediately decrease ? or need to do some other action to refresh it ?

Re: problem with yarn from ambari dashboard

jsensharma — Wed, 25 Oct 2017 16:06:43 GMT

@uri ben-ari

I will have to check with some Yarn experts. I will check and update.

Re: problem with yarn from ambari dashboard

mike_bronson7 — Wed, 25 Oct 2017 16:08:48 GMT

ok thank you waiting for your answer

Re: problem with yarn from ambari dashboard

jsensharma — Wed, 25 Oct 2017 16:18:14 GMT

@uri ben-ari

This is the response i got from @pjoseph

Example:

- The "yarn.nodemanager.resource.memory-mb" is NodeManager memory reserved for yarn tasks.
- So for example, the machine RAM is 300GB and you reserve 50GB for OS, Dameons and provide 250GB to yarn
- So i configure "yarn.nodemanager.resource.memory-mb" to 250GB
- Say i have 5 nodes, each node will register to RM once at startup
- So RM will have total memory 250 * 5 = 1250GB

- In case m increasing the yarn.nodemanager.resource.memory-mb value to 260GB and restarting NM now, 260 * 5 will be total

Re: problem with yarn from ambari dashboard

mike_bronson7 — Wed, 25 Oct 2017 16:18:48 GMT

I update the question with more info ( like which application are runing under yarn and machine memory )

Re: problem with yarn from ambari dashboard

jsensharma — Wed, 25 Oct 2017 16:22:36 GMT

@uri ben-ari

You can also have a look at the "The HDP utility script is the recommended method for calculating HDP memory configuration settings"

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_command-line-installation/content/determine-hdp-memory-config.html

Re: problem with yarn from ambari dashboard

mike_bronson7 — Wed, 25 Oct 2017 16:27:30 GMT

so on each worker machine we have 32G ( lets say we allocate 7 to the OS ) so we have 25G , in my system we have 5 workers so this mean 25*5=125 , and this is actually what configured , am I corect ? ( we have in the cluster 3 master machines , 5 workers machines ) , in that case we need to increase each worker memory to at least 50G - am I correct ? and after that set the value of yarn.nodemanager.resource.memory-mb to 260G for example

Re: problem with yarn from ambari dashboard

mike_bronson7 — Wed, 25 Oct 2017 16:58:50 GMT

Jay not in this issue , but I will happy to get tour answer about my quastion from - https://community.hortonworks.com/questions/142356/how-to-recover-the-standby-name-node-in-ambari-clu.html