question Re: The heap memory usage of NameNode is much higher than expected in Support Questions

The heap memory usage of NameNode is much higher than expected

Meepoljd — Tue, 16 Jan 2024 08:52:22 GMT

Hi, I have a Hadoop 3.1.1 cluster and recently I found that the NameNode heap memory usage in the cluster is very high. I saw 629973631 file objects in the cluster through WebUi, so according to my calculations, it should occupy no more than 90GB of memory, right? Why is the current memory usage consistently above 140GB? Is this related to me enabling erasure codes?

Re: The heap memory usage of NameNode is much higher than expected

mszurap — Wed, 17 Jan 2024 09:07:54 GMT

Hi @Meepoljd ,

The file and block metadata consumes the NameNode heap. Can you share how did your calculation?

Per our docs:

https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/hdfs-overview/topics/hdfs-sizing-namenode-heap-memory.html

the file count should kept below 300m files. Also the same page suggests that approximately 150 bytes are needed for each namespace object, I assume you did your calculation based on that. The real NN heap consumption varies with the path lengths, ACL counts, replication factors, snapshots, operational load, etc. As such in our other page

https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/hdfs-overview/topics/hdfs-examples-namenode-heap-memory.html

we suggest to allocate rather a bigger heap size, 1 GB heap for 1 million blocks, which would be ~320 GB in your case.

Hope this helps,

Best regards, Miklos

Re: The heap memory usage of NameNode is much higher than expected

Meepoljd — Thu, 18 Jan 2024 08:48:21 GMT

Thank you very much for your answer. I will try to adjust the allocation of heap memory. In addition, I would like to ask how the conclusion of using 1GB of memory for every 1 million blocks was drawn? Or is there a more precise calculation method that can lead to such a conclusion?

Re: The heap memory usage of NameNode is much higher than expected

Meepoljd — Thu, 18 Jan 2024 08:48:44 GMT

Re: The heap memory usage of NameNode is much higher than expected

mszurap — Thu, 18 Jan 2024 13:52:13 GMT

As far as I know this is more of an empirical best practice. As mentioned it cannot be exactly calculated since there are some variable factors (filename / path lengths, acl counts, etc) which change from environment to environment.