Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What's the problem on having dfs.namenode.handler.count configured too high?

Solved Go to solution
Highlighted

What's the problem on having dfs.namenode.handler.count configured too high?

Rising Star

I am following Smartsense recommendation to make my cluster operationally more efficient, however there is a lot of documentation on how to set mentioned feature within ambari an why handlers are important for RPC, no one is mentioning why having this feature set "too high" affects a cluster.

Can anyone share some thought here?

Regards!"

1 ACCEPTED SOLUTION

Accepted Solutions

Re: What's the problem on having dfs.namenode.handler.count configured too high?

Rising Star

Below the answer in case someone need it:

"Each handler (worker) thread consume resources/memory on the NN. We can not set this value too high because it will consume unnecessary resources and cause extra burden on the NN."

2 REPLIES 2

Re: What's the problem on having dfs.namenode.handler.count configured too high?

Rising Star

Below the answer in case someone need it:

"Each handler (worker) thread consume resources/memory on the NN. We can not set this value too high because it will consume unnecessary resources and cause extra burden on the NN."

Re: What's the problem on having dfs.namenode.handler.count configured too high?

New Contributor

It is recommended that the RPC handler count is set to 20 * log2(Cluster Size) with an upper limit of 200. Refer more details here : https://community.hortonworks.com/articles/43839/scaling-the-hdfs-namenode-part-2.html

Don't have an account?
Coming from Hortonworks? Activate your account here