Thanks for the post, made me re-examine my assumptions based on available topics that come up on a search. The number i had always heard is 150 bytes for every block, file, and directory. [1][2] this combined with a heavy dose of "back of the envelope" type math and a taking it all times 2 as a safety factor has worked well for me in the past for a quick estimation of the amount of jvm heap you need. [1]http://blog.cloudera.com/blog/2009/02/the-small-files-problem/ [2]http://www.mail-archive.com/core-user@hadoop.apache.org/msg02835.html If the above is actually true then the size of the files, block size, and number of directories all matter for your raw heap needed calculations. keep in mind all of the handlers use heap as well, so this won't be 100% accurate for predicting namenode heap necessary for x many files, directories, and blocks. If you are interested in performing an experiment to find out what it really is, not just what it is theoritically by looking at the individual methods, I think you could find this by starting a test cluster, take a heap dump, write a single file less than the block size to HDFS / with replication 1, take another heap dump, and then compare. then you can repeat the experiment each variable, (i.e. create the file in a directory, create a file bigger than 1 block, create a file with more than 1 replica). good luck and let us all know what you find! -Ben
... View more
