Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Namenode Heap Size usage is 80 to 86 % on daily basis

avatar
Expert Contributor

80GB is allocated and 67.4 GB utilized

HDFS Max Alloted : 860 TB

HDFS Data use is : 500TB

Block Size : 128 MB

Could you help me to address the NameNode Heap Memory Use.

1 ACCEPTED SOLUTION

avatar
Master Guru
@zkfs

What version of HDP are you using? NN on older HDP versions has some issues while rendering blockstate related stuff in logs.

Please have a look at your hdfs-audit logs to see number of operations per second.

Also, please check below documentation to set recommended heap size according to total number of files:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/ref-809...

View solution in original post

5 REPLIES 5

avatar
Master Guru
@zkfs

What version of HDP are you using? NN on older HDP versions has some issues while rendering blockstate related stuff in logs.

Please have a look at your hdfs-audit logs to see number of operations per second.

Also, please check below documentation to set recommended heap size according to total number of files:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/ref-809...

avatar
Expert Contributor

I am using version as below

Cluster Stack Version: HDP-2.1

Ambari 1.6 version

HDFS 2.4

avatar
Expert Contributor

What are other main factor for namenode heap size is getting utilized for in cluster

avatar
Super Guru

@zkfs

Just because your block is set to 128 MB doesn't mean you don't have small files. Please use fsck to find out more details about your filesystem. It is likely that you have a lot of small files.

avatar
Expert Contributor

If small files are there in HDFS storage is it going to occupy the Namenode Heap Memory ?

1) What is work around for total Name Node Memory Heap size usage.

2) Is any other things need to consider, YARN jobs ...etc

your concerns are welcome.