Is the Cloudera documentation on Heap calculation correct ?

[#1]

I was refering to the documentation at: http://www.cloudera.com/documentation/enterprise/latest/topics/admin_nn_memory_config.html

But, it says that the replication factor does not impact heap size on the namenode, which does not seem correct as the data structure uses about 16 bytes for each replica. The class BlockInfo has a Datanode Descriptor that maps the nodes on which the replica's of the block are present. or to be more precise block_files of a block.

So, if I have 1 million files, with each file having 2 blocks per file, the calculations will be approximately as below:

Memory Requirements:

1 million files = 150 bytes x 1 million = 143.0511474609375 MB

6 millions blocks( replication 3) = (150 bytes per block + 16 bytes per replica = 150 bytes + 2 x16 = 182 bytes)x 6 million = 1041.412353515625 MB

Total = 1.15670259132385 GB

Am i correct or I am missing something ?

[#2]

Can someone please look into it.

[#3]

Thanks for the post, made me re-examine my assumptions based on available topics that come up on a search.

The number i had always heard is 150 bytes for every block, file, and directory. [1][2] this combined with a heavy dose of "back of the envelope" type math and a taking it all times 2 as a safety factor has worked well for me in the past for a quick estimation of the amount of jvm heap you need. 

[1]http://blog.cloudera.com/blog/2009/02/the-small-files-problem/

[2]http://www.mail-archive.com/core-user@hadoop.apache.org/msg02835.html

If the above is actually true then the size of the files, block size, and number of directories all matter for your raw heap needed calculations. keep in mind all of the handlers use heap as well, so this won't be 100% accurate for predicting namenode heap necessary for x many files, directories, and blocks. 

If you are interested in performing an experiment to find out what it really is, not just what it is theoritically by looking at the individual methods, I think you could find this by starting a test cluster, take a heap dump, write a single file less than the block size to HDFS / with replication 1, take another heap dump, and then compare. then you can repeat the experiment each variable, (i.e. create the file in a directory, create a file bigger than 1 block, create a file with more than 1 replica).

good luck and let us all know what you find!

-Ben

[#4]

Thanks Ben, I agree to your point on performance ascepts that there are other things we consider while calculating this and it is a rough estimate. In production there are other factor which dictate heap of namenode like handler count, how many Datanodes can send block report in a given time frame(initDelay - parameter) and few others.

Yes, I have tested that blocks, files, directories everything counts.

I brought this up for the sake of hadoop learners that the statement "replication does not impact memory on Namenode" is incorrect.

Thanks and Regards

[#5 · Accepted solution]

Ah I see, then yes I agree, replication does impact fsimage and memory footprint and that documentation should be updated to reflect that.