Namenode Heap Size usage is 80 to 86 % on daily basis

[#1]

80GB is allocated and 67.4 GB utilized

HDFS Max Alloted : 860 TB

HDFS Data use is : 500TB

Block Size : 128 MB

Could you help me to address the NameNode Heap Memory Use.

[#2 · Accepted solution]

@zkfs

What version of HDP are you using? NN on older HDP versions has some issues while rendering blockstate related stuff in logs.

Please have a look at your hdfs-audit logs to see number of operations per second.

Also, please check below documentation to set recommended heap size according to total number of files:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/ref-809...

[#3]

I am using version as below

Cluster Stack Version: HDP-2.1

Ambari 1.6 version

HDFS 2.4

[#4]

What are other main factor for namenode heap size is getting utilized for in cluster

[#5]

@zkfs

Just because your block is set to 128 MB doesn't mean you don't have small files. Please use fsck to find out more details about your filesystem. It is likely that you have a lot of small files.

[#6]

If small files are there in HDFS storage is it going to occupy the Namenode Heap Memory ?

1) What is work around for total Name Node Memory Heap size usage.

2) Is any other things need to consider, YARN jobs ...etc

your concerns are welcome.