NameNode heapsize is bigger then it should be.

[#1]

Hi,

I am running HDP-2.3.4.0 and Ambari 2.2.0.0

The NameNode heap size is set to 24Gb where 22Gb are used. This is very strange since the total blocks number is 7,796,546.

The fsimage size is 19Gb. Here is the jmap output for namenode JVM

-bash-4.2$ /usr/java/jdk1.7.0_67/bin/jmap -histo 35350 | head

#num #instances #bytes class name

----------------------------------------------

1: 175738748 14270581184 [B

2: 193636970 13941861840 org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy

3: 193636970 10843670320 org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff

4: 15675953 2867785824 [Ljava.lang.Object;

5: 4 1761607776 [Lorg.apache.hadoop.util.LightWeightGSet$LinkedElement;

6: 7798931 748697376 org.apache.hadoop.hdfs.server.namenode.INodeFile

7: 7799784 499186176 org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguous

According to the best practices, we could use about 9 or 10Gb heap size. But this is not the case.

What could impact the heap size and how could I troubleshot this?

[#2]

It is likely the process has not hit an allocation failure yet so GC has not kicked in. This is perfectly normal. If you want the heap usage to be lower then you can reduce the heap allocation.

Alternatively you can trigger GC quicker by adding something like -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly to your heap options.

However it's probably best to just follow our suggested heap configuration and let the Java runtime do the rest.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_command-line-installation/content/config...

[#3]

I feel this is not a case, since fsimage size is 19Gb for about 8,000,000 blocks - which is too much. I have updated the question with more info.

[#4 · Accepted solution]

Try clearing up some snapshots. You probably have a ton of deleted files retained for snapshots.

[#5]

Thank you! Yes, we have a lot of snapshots.