Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is there any recommendation for dfs.datanode.max.transfer.thread ?

Highlighted

Is there any recommendation for dfs.datanode.max.transfer.thread ?

Super Guru

I know that default value is 4096. As per Hortonworks recommendation, we need to set it to 16K if we are using HBase.

I think we should consider number of cores, main memory, ulimits etc. I need to understand how can we decide best value of this property?

http://blog.cloudera.com/blog/2012/03/hbase-hadoop-xceivers/ has some formulas which we can use for deciding value.

Please advice! :)

1 REPLY 1

Re: Is there any recommendation for dfs.datanode.max.transfer.thread ?

Guru

There is a longer version of that post here. http://www.larsgeorge.com/2012/03/hadoop-hbase-and-xceivers.html

You can use the formula at the end of it.

3791-hadoophbasexceiverformula4.jpg

I think Hortonworks recommendation is taking the easier approach and putting a higher limit which works in most cases.

Don't have an account?
Coming from Hortonworks? Activate your account here