Support Questions
Find answers, ask questions, and share your expertise

What is the max latency (ms) acceptable between HDFS data nodes?​

What is the max latency (ms) acceptable between HDFS data nodes?​

What is the max latency (ms) acceptable between HDFS data nodes?

7 REPLIES 7

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

@Sean Roberts

For a dedicated network, anything under 10 ms is probably acceptable (anything > 6 ms with a grim on my face), however, preferred is 2-4 ms. This is just personal experience.

A good resource for measuring the latency: https://community.hortonworks.com/articles/12895/measuring-network-latency-between-nodes.html

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

Hi @Sean Roberts, HDFS works best with short inter-node latencies. Generally nodes are in the same data center and inter-node latency is within 1 millisecond.

Do you mind describing the use case you have in mind?

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

Rising Star

300ms is the default threshold for generating WARN message in datanode log like below:

[Timestamp] WARN org.apache.hadoop.hdfs.server.datanode.DataNode:Slow BlockReceiver write packet to mirror took 350ms (threshold=300ms)

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

@Xiaoyu Yao - Thanks. Do you know if that settings is adjustable and do you know a good reference for why that number was chosen?

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

@Xiaoyu Yao - Are you surethat is for the network?

It appears to be tied to this setting which says "io" in it: `dfs.datanode.slow.io.warning.threshold.ms`

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

Rising Star

@Sean Roberts, thanks for the correction. Yes, that's is a threshold for network bandwidth, i.e. time spent in copying a packet (64K) from one datanode to another.

Re: What is the max latency (ms) acceptable between HDFS data nodes?​

@Xiaoyu Yao - Can you update your original answer with that detail and delete this separate answer? I can accept it then.