Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What is the max latency (ms) acceptable between HDFS data nodes?​

Highlighted

What is the max latency (ms) acceptable between HDFS data nodes?​

What is the max latency (ms) acceptable between HDFS data nodes?

7 REPLIES 7
Highlighted

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

@Sean Roberts

For a dedicated network, anything under 10 ms is probably acceptable (anything > 6 ms with a grim on my face), however, preferred is 2-4 ms. This is just personal experience.

A good resource for measuring the latency: https://community.hortonworks.com/articles/12895/measuring-network-latency-between-nodes.html

Highlighted

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

Hi @Sean Roberts, HDFS works best with short inter-node latencies. Generally nodes are in the same data center and inter-node latency is within 1 millisecond.

Do you mind describing the use case you have in mind?

Highlighted

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

Rising Star

300ms is the default threshold for generating WARN message in datanode log like below:

[Timestamp] WARN org.apache.hadoop.hdfs.server.datanode.DataNode:Slow BlockReceiver write packet to mirror took 350ms (threshold=300ms)

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

@Xiaoyu Yao - Thanks. Do you know if that settings is adjustable and do you know a good reference for why that number was chosen?

Highlighted

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

@Xiaoyu Yao - Are you surethat is for the network?

It appears to be tied to this setting which says "io" in it: `dfs.datanode.slow.io.warning.threshold.ms`

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

Highlighted

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

Rising Star

@Sean Roberts, thanks for the correction. Yes, that's is a threshold for network bandwidth, i.e. time spent in copying a packet (64K) from one datanode to another.

Highlighted

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

@Xiaoyu Yao - Can you update your original answer with that detail and delete this separate answer? I can accept it then.

Don't have an account?
Coming from Hortonworks? Activate your account here