What is the max latency (ms) acceptable between HDFS data nodes?
For a dedicated network, anything under 10 ms is probably acceptable (anything > 6 ms with a grim on my face), however, preferred is 2-4 ms. This is just personal experience.
A good resource for measuring the latency: https://community.hortonworks.com/articles/12895/measuring-network-latency-between-nodes.html
Hi @Sean Roberts, HDFS works best with short inter-node latencies. Generally nodes are in the same data center and inter-node latency is within 1 millisecond.
Do you mind describing the use case you have in mind?
300ms is the default threshold for generating WARN message in datanode log like below:
[Timestamp] WARN org.apache.hadoop.hdfs.server.datanode.DataNode:Slow BlockReceiver write packet to mirror took 350ms (threshold=300ms)
@Xiaoyu Yao - Are you surethat is for the network?
It appears to be tied to this setting which says "io" in it: `dfs.datanode.slow.io.warning.threshold.ms`