question Re: NameNode heapsize is bigger then it should be. in Support Questions

NameNode heapsize is bigger then it should be.

Oleg_Hmelnits — Thu, 26 Oct 2017 22:59:32 GMT

Hi,

I am running HDP-2.3.4.0 and Ambari 2.2.0.0

The NameNode heap size is set to 24Gb where 22Gb are used. This is very strange since the total blocks number is 7,796,546.

The fsimage size is 19Gb. Here is the jmap output for namenode JVM

-bash-4.2$ /usr/java/jdk1.7.0_67/bin/jmap -histo 35350 | head

#num #instances #bytes class name

----------------------------------------------

1: 175738748 14270581184 [B

2: 193636970 13941861840 org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy

3: 193636970 10843670320 org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff

4: 15675953 2867785824 [Ljava.lang.Object;

5: 4 1761607776 [Lorg.apache.hadoop.util.LightWeightGSet$LinkedElement;

6: 7798931 748697376 org.apache.hadoop.hdfs.server.namenode.INodeFile

7: 7799784 499186176 org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguous

According to the best practices, we could use about 9 or 10Gb heap size. But this is not the case.

What could impact the heap size and how could I troubleshot this?

Re: NameNode heapsize is bigger then it should be.

ArpitAgarwal — Fri, 27 Oct 2017 02:12:15 GMT

It is likely the process has not hit an allocation failure yet so GC has not kicked in. This is perfectly normal. If you want the heap usage to be lower then you can reduce the heap allocation.

Alternatively you can trigger GC quicker by adding something like -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly to your heap options.

However it's probably best to just follow our suggested heap configuration and let the Java runtime do the rest.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_command-line-installation/content/configuring-namenode-heap-size.html

Re: NameNode heapsize is bigger then it should be.

Oleg_Hmelnits — Fri, 27 Oct 2017 02:41:21 GMT

I feel this is not a case, since fsimage size is 19Gb for about 8,000,000 blocks - which is too much. I have updated the question with more info.

Re: NameNode heapsize is bigger then it should be.

ArpitAgarwal — Sat, 28 Oct 2017 01:16:04 GMT

Try clearing up some snapshots. You probably have a ton of deleted files retained for snapshots.

Re: NameNode heapsize is bigger then it should be.

Oleg_Hmelnits — Sat, 28 Oct 2017 01:19:29 GMT

Thank you! Yes, we have a lot of snapshots.