Support Questions
Find answers, ask questions, and share your expertise

Why my cluster memory is less even though physical memory is more..?

Solved Go to solution
Highlighted

Why my cluster memory is less even though physical memory is more..?

Explorer

We have 2-node cluster(1 master 4 CPU,16 GB RAM + 1 data node 8 CPU,30 GB RAM). However in Ambari console, I could be able to see the Total cluster memory is 22 GB only. Is there a way to allocate more cluster memory(around 36GB ) out of 46 GB physical memory we have together from master + data node. Morever, the number of containers are only 5 whereas the available Vcores are 8 already. I have attached the screenshot for your reference. Please suggest a way to improve the cluster resource utilization. Thank you in advance.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Why my cluster memory is less even though physical memory is more..?

Rising Star

@Saravanan Ramaraj

I assume the question is around the YARN total memory.

This is because Ambari uses the smallest capacity node to bring with the calculations, as Ambari expects a homogenous cluster.

But in this case, we have heterogenous cluster as : 1 master 4 CPU,16 GB RAM + 1 data node 8 CPU,30 GB RAM

- Thus, Ambari picks the 16 GB one and assumes 2nd one to be of same size and does the calculation for YARN's Node Manager (NM) memory. I assume that both nodes have Node Manager running.

- I believe that you would have 11 GB as value for YARN/yarn.nodemanager.resource.memory-mb. Thus, we have 22 GB (11 * 2) available in this case which is > 16 GB. 16*2 = 32 GB, but Ambari takes out memory required to run other processes outside the YARN workspace (eg: RM, HBase etc). Thus we have memory less than 32 GB available (which is expected).

Its a good idea to have homogeneous clusters.

===================================================================

However, you can make use of Config Groups here in Ambari based on different hardware profiles.

You can creates 2 Config Groups (CG) where each CG has one node. By default, there would be a default CG as seen on YARN configs page having both the nodes.

How to create a CG is exemplified using HBase here : https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_Ambari_Users_Guide/content/_using_host_c...

I did the following testing in order to reduce the memory for one node. You can similarly bump up up the memory for the 30 GB node.

- Starting with 2 node cluster, where Ambari had given 12 GB to each NM, with total capacity being 24 GB.

39984-screen-shot-2017-10-26-at-111536-pm.png

- Created a CG named 'New' and added 2nd node to it. Then changed the YARN/yarn.nodemanager.resource.memory-mb for 2nd node under 'New' from ~ 12 GB to ~8 GB.

39985-screen-shot-2017-10-26-at-103835-pm.png

- State of Node 1 under 'default' CG:

39986-screen-shot-2017-10-26-at-103845-pm.png

- Restarted "Affected components" as prompted by Ambari after the above changes.

- The Total memory changes from 24 GB to 20 GB now.

39987-screen-shot-2017-10-26-at-113313-pm.png

Hope this helps.

View solution in original post

4 REPLIES 4

Re: Why my cluster memory is less even though physical memory is more..?

@Saravanan Ramaraj

Ideally Ambari should show your total RAM and CPU information, until if you have any issues with Ambari agent.

Can you provide free -m output from your both the nodes and add scree shots of ambari cluster information.??

Highlighted

Re: Why my cluster memory is less even though physical memory is more..?

Rising Star

@Saravanan Ramaraj

I assume the question is around the YARN total memory.

This is because Ambari uses the smallest capacity node to bring with the calculations, as Ambari expects a homogenous cluster.

But in this case, we have heterogenous cluster as : 1 master 4 CPU,16 GB RAM + 1 data node 8 CPU,30 GB RAM

- Thus, Ambari picks the 16 GB one and assumes 2nd one to be of same size and does the calculation for YARN's Node Manager (NM) memory. I assume that both nodes have Node Manager running.

- I believe that you would have 11 GB as value for YARN/yarn.nodemanager.resource.memory-mb. Thus, we have 22 GB (11 * 2) available in this case which is > 16 GB. 16*2 = 32 GB, but Ambari takes out memory required to run other processes outside the YARN workspace (eg: RM, HBase etc). Thus we have memory less than 32 GB available (which is expected).

Its a good idea to have homogeneous clusters.

===================================================================

However, you can make use of Config Groups here in Ambari based on different hardware profiles.

You can creates 2 Config Groups (CG) where each CG has one node. By default, there would be a default CG as seen on YARN configs page having both the nodes.

How to create a CG is exemplified using HBase here : https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_Ambari_Users_Guide/content/_using_host_c...

I did the following testing in order to reduce the memory for one node. You can similarly bump up up the memory for the 30 GB node.

- Starting with 2 node cluster, where Ambari had given 12 GB to each NM, with total capacity being 24 GB.

39984-screen-shot-2017-10-26-at-111536-pm.png

- Created a CG named 'New' and added 2nd node to it. Then changed the YARN/yarn.nodemanager.resource.memory-mb for 2nd node under 'New' from ~ 12 GB to ~8 GB.

39985-screen-shot-2017-10-26-at-103835-pm.png

- State of Node 1 under 'default' CG:

39986-screen-shot-2017-10-26-at-103845-pm.png

- Restarted "Affected components" as prompted by Ambari after the above changes.

- The Total memory changes from 24 GB to 20 GB now.

39987-screen-shot-2017-10-26-at-113313-pm.png

Hope this helps.

View solution in original post

Highlighted

Re: Why my cluster memory is less even though physical memory is more..?

Explorer

Thanks much for your detailed reply, it really helps.!!!

Highlighted

Re: Why my cluster memory is less even though physical memory is more..?

Rising Star

Thanks. Glad to know that it helped.