Support Questions
Find answers, ask questions, and share your expertise

High NameNode Heap Usage

High NameNode Heap Usage

Contributor

NameNode is running at 75% to 85% heap usage even when the cluster is at near idle state.

As per the following link, we should set the name node size at 1gb and Young Generation Size should be 128mb but we have set the heap size at 3gb and Young Generation Size at 384mb.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/ref-8095...

Should i change it to 1gb and 128mb as per the Hortonworks recommendation? or leave the settings as is and change some other parameter to bring the usage down? Please advise

Some other info

Total HDFS Data Size < 1 TB

Number of Files < 160k

Block Size 128 mb

HDP 2.3.2

7 REPLIES 7

Re: High NameNode Heap Usage

@rbalam

Generally, Ambari does good job in choosing the settings. If you are not facing any issues then I won't change it otherwise, you can go with the official docs.

Re: High NameNode Heap Usage

Contributor

The settings we currently have are recommended by Ambari itself, but the client is asking why the namenode usage is so high when there is no load on it. does it make sense to increase the heapsize to 4gb to bring the usage percentage down? or we will run into issues if we increase from 3gb to 4gb? please adivse

Re: High NameNode Heap Usage

@rbalam You can increase to 3GB

Re: High NameNode Heap Usage

Usually if you have less than a million files you can go with 1024MB RAM for the Namenode (128MB -XX:NewSize -XX:MaxNewSize). You can also increase it to 3GB to be on the safe side, the Young Generation size should be 1/8 of the NN heap size.

What does your hardware setup look like? (VM or bare metal?) Whats the total memory that is available on the NN machine?

I have seen a similar heap usage on one of my test clusters. I had 1GB heap configured, about 15K blocks and the heap usage was at 25%. After restarting the Namenode the heap usage dropped to 6%.

Re: High NameNode Heap Usage

Contributor

@Jonas Straub

This is not a VM and it has 256gb RAM and 64cores CPU.

We already have 3gb for Name Node heap and 384mb young generation size. So I am having tough time explaining to the customer on why the heap usage is high when there is no load. What takes the name node heap to that high when we have < 120k files and 200gb hdfs data with 128mb block size?

and also we tried to restart the name node to solve this problem. It helped to bring the heap size down but it went back to >75% after a day.

Re: High NameNode Heap Usage

Mentor

@rbalam @Chris Nauroth posted a great article just today about this.

Re: High NameNode Heap Usage

Rising Star

@rbalam

I'm facing the same problem with ambari 2.4.1.0 version.

For the moment, i increase the memory size to 3Gb. Do you get the real reason behind this? It'is a normal behaviour of recent version of ambari? thanks in advance.