question Re: Namenode Best Pratices / Performance tunning checklist in Archives of Support Questions (Read Only)

Namenode Best Pratices / Performance tunning checklist

gbraccialli3 — Mon, 07 Dec 2015 19:44:03 GMT

Does anyone have a best practices guide / performance checklist for namenode?

I know when we increase namenode_heapsize, we should also adjust namenode_opt_newsize, namenode_opt_maxnewsize, namenode_opt_permsize, namenode_opt_maxpermsize, but what are the reference calculations for this adjustment?

Also there is: dfs.namenode.handler.count and dfs.namenode.service.handler.count, as discussed in question below, but same here, what are reference values for these properties? Is it based number of nodes only or should also be increased based on usage patterns? https://community.hortonworks.com/questions/4799/is-there-any-suggested-ratio-for-dividing-the-numb.html#answer-5206

Re: Namenode Best Pratices / Performance tunning checklist

rafal_wabinski — Thu, 10 Dec 2015 05:29:35 GMT

Informations about namenode_heapsize are in documentation:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/determine-hdp-memory-config.html

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-80953924-1cbf-4655-9953-1e744290a6c3.1.html

Re: Namenode Best Pratices / Performance tunning checklist

kmungee — Thu, 10 Dec 2015 05:37:27 GMT

-XX:NewSize and - XX:MaxnewSize should be 1/8 of the maximum heap size (-Xmx). So if -Xmx is set to 8GB, the --XX:NewSize=- XX:MaxnewSize should be set to 1GB.

The value of dfs.namenode.handler.count is calculated based on the number of datanodes in the cluster. @Arpit Agarwal suggested that the value for this should be ln(number of datanodes)*20. For example in a 450 node cluster it can be set to around 180.

Thanks,

Koelli

Re: Namenode Best Pratices / Performance tunning checklist

gbraccialli3 — Thu, 10 Dec 2015 10:30:21 GMT

Thank you @kmungee.

Would you have other configuration to check on namenode service?