Namenode Best Pratices / Performance tunning checklist

[#1]

Does anyone have a best practices guide / performance checklist for namenode?

I know when we increase namenode_heapsize, we should also adjust namenode_opt_newsize, namenode_opt_maxnewsize, namenode_opt_permsize, namenode_opt_maxpermsize, but what are the reference calculations for this adjustment?

Also there is: dfs.namenode.handler.count and dfs.namenode.service.handler.count, as discussed in question below, but same here, what are reference values for these properties? Is it based number of nodes only or should also be increased based on usage patterns? https://community.hortonworks.com/questions/4799/is-there-any-suggested-ratio-for-dividing-the-numb....

[#2]

Informations about namenode_heapsize are in documentation:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/determin...

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-8095...

[#3 · Accepted solution]

-XX:NewSize and - XX:MaxnewSize should be 1/8 of the maximum heap size (-Xmx). So if -Xmx is set to 8GB, the --XX:NewSize=- XX:MaxnewSize should be set to 1GB.

The value of dfs.namenode.handler.count is calculated based on the number of datanodes in the cluster. @Arpit Agarwal suggested that the value for this should be ln(number of datanodes)*20. For example in a 450 node cluster it can be set to around 180.

Thanks,

Koelli

[#4]

Thank you @kmungee.

Would you have other configuration to check on namenode service?