question Re: What's the problem on having dfs.namenode.handler.count configured too high? in Archives of Support Questions (Read Only)

What's the problem on having dfs.namenode.handler.count configured too high?

JLo_Hernandez — Wed, 12 Sep 2018 22:43:45 GMT

I am following Smartsense recommendation to make my cluster operationally more efficient, however there is a lot of documentation on how to set mentioned feature within ambari an why handlers are important for RPC, no one is mentioning why having this feature set "too high" affects a cluster.

Can anyone share some thought here?

Regards!"

Re: What's the problem on having dfs.namenode.handler.count configured too high?

JLo_Hernandez — Fri, 14 Sep 2018 01:39:17 GMT

Below the answer in case someone need it:

"Each handler (worker) thread consume resources/memory on the NN. We can not set this value too high because it will consume unnecessary resources and cause extra burden on the NN."

Re: What's the problem on having dfs.namenode.handler.count configured too high?

apathak — Fri, 14 Sep 2018 01:44:00 GMT

It is recommended that the RPC handler count is set to 20 * log2(Cluster Size) with an upper limit of 200. Refer more details here : https://community.hortonworks.com/articles/43839/scaling-the-hdfs-namenode-part-2.html