Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

What's the problem on having dfs.namenode.handler.count configured too high?

Rising Star

I am following Smartsense recommendation to make my cluster operationally more efficient, however there is a lot of documentation on how to set mentioned feature within ambari an why handlers are important for RPC, no one is mentioning why having this feature set "too high" affects a cluster.

Can anyone share some thought here?

Regards!"

1 ACCEPTED SOLUTION

Rising Star

Below the answer in case someone need it:

"Each handler (worker) thread consume resources/memory on the NN. We can not set this value too high because it will consume unnecessary resources and cause extra burden on the NN."

View solution in original post

2 REPLIES 2

Rising Star

Below the answer in case someone need it:

"Each handler (worker) thread consume resources/memory on the NN. We can not set this value too high because it will consume unnecessary resources and cause extra burden on the NN."

Cloudera Employee

It is recommended that the RPC handler count is set to 20 * log2(Cluster Size) with an upper limit of 200. Refer more details here : https://community.hortonworks.com/articles/43839/scaling-the-hdfs-namenode-part-2.html