question Namenode Heap Size usage is 80 to 86 % on daily basis in Archives of Support Questions (Read Only)

Namenode Heap Size usage is 80 to 86 % on daily basis

shivkumar82015 — Thu, 11 Aug 2016 12:48:04 GMT

80GB is allocated and 67.4 GB utilized

HDFS Max Alloted : 860 TB

HDFS Data use is : 500TB

Block Size : 128 MB

Could you help me to address the NameNode Heap Memory Use.

Re: Namenode Heap Size usage is 80 to 86 % on daily basis

KuldeepK — Thu, 11 Aug 2016 13:29:10 GMT

@zkfs

What version of HDP are you using? NN on older HDP versions has some issues while rendering blockstate related stuff in logs.

Please have a look at your hdfs-audit logs to see number of operations per second.

Also, please check below documentation to set recommended heap size according to total number of files:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/ref-80953924-1cbf-4655-9953-1e744290a6c3.1.html

Re: Namenode Heap Size usage is 80 to 86 % on daily basis

shivkumar82015 — Fri, 12 Aug 2016 15:07:01 GMT

I am using version as below

Cluster Stack Version: HDP-2.1

Ambari 1.6 version

HDFS 2.4

Re: Namenode Heap Size usage is 80 to 86 % on daily basis

shivkumar82015 — Sun, 14 Aug 2016 21:51:06 GMT

What are other main factor for namenode heap size is getting utilized for in cluster

Re: Namenode Heap Size usage is 80 to 86 % on daily basis

mqureshi — Mon, 15 Aug 2016 02:19:52 GMT

@zkfs

Just because your block is set to 128 MB doesn't mean you don't have small files. Please use fsck to find out more details about your filesystem. It is likely that you have a lot of small files.

Re: Namenode Heap Size usage is 80 to 86 % on daily basis

shivkumar82015 — Thu, 18 Aug 2016 19:11:54 GMT

If small files are there in HDFS storage is it going to occupy the Namenode Heap Memory ?

1) What is work around for total Name Node Memory Heap size usage.

2) Is any other things need to consider, YARN jobs ...etc

your concerns are welcome.