question Re: How to identify stale datanode? in Archives of Support Questions (Read Only)

How to identify stale datanode?

ayusuf — Fri, 30 Oct 2015 07:38:22 GMT

Datanode Health Summary in Ambari Alerts reported 1 stale node. How to identify which datannode is in stale state?

Re: How to identify stale datanode?

ssingla — Fri, 30 Oct 2015 07:38:49 GMT

probably the namenode logs should say that..

Re: How to identify stale datanode?

nsabharwal — Fri, 30 Oct 2015 08:00:12 GMT

@ayusuf@hortonworks.com

This is good explanation ...namenode will know about the stale DN

dfs.namenode.stale.datanode.interval

Default time interval for marking a datanode as "stale", i.e., if the namenode has not received heartbeat msg from a datanode for more than this time interval, the datanode will be marked and treated as "stale" by default. The stale interval cannot be too small since otherwise this may cause too frequent change of stale states. We thus set a minimum stale interval value (the default value is 3 times of heartbeat interval) and guarantee that the stale interval cannot be less than the minimum value. A stale data node is avoided during lease/block recovery. It can be conditionally avoided for reads (see dfs.namenode.avoid.read.stale.datanode) and for writes (see dfs.namenode.avoid.write.stale.datanode).

Re: How to identify stale datanode?

amiller — Fri, 30 Oct 2015 11:13:18 GMT

A datanode is considered stale when:

dfs.namenode.stale.datanode.interval < last contact < (2 * dfs.namenode.heartbeat.recheck-interval)

In the NameNode UI Datanodes tab, a stale datanode will stand out due to having a larger value for Last contact among live datanodes (also available in JMX output). When a datanode is stale, it will be given lowest priority for reads and writes.

Using default values, the namenode will consider a datanode stale when its heartbeat is absent for 30 seconds. After another 10 minutes without a heartbeat (10.5 minutes total), a datanode is considered dead.

Relevant properties include:

dfs.heartbeat.interval - default: 3 seconds
dfs.namenode.stale.datanode.interval - default: 30 seconds
dfs.namenode.heartbeat.recheck-interval - default: 5 minutes
dfs.namenode.avoid.read.stale.datanode - default: true
dfs.namenode.avoid.write.stale.datanode - default: true

This feature was introduced by HDFS-3703.

Re: How to identify stale datanode?

nsabharwal — Fri, 30 Oct 2015 16:43:04 GMT

Nicely explained! Thanks @Alex Miller

Re: How to identify stale datanode?

ayusuf — Sat, 31 Oct 2015 00:31:36 GMT

Thanks Alex. Very good explanation. 😞 I learned this the hard way yesterday night, by bringing the network down (ifdown eth1) while the datanode was up in one of VM nodes and refreshing the Namenode UI -> Datanode tab. @Alex Miller

Re: How to identify stale datanode?

amiller — Sat, 31 Oct 2015 01:22:54 GMT

Thanks, hopefully it will save someone the hassle in the future.

In the future, please leave this as a comment rather than a separate answer.

Re: How to identify stale datanode?

ayusuf — Sat, 31 Oct 2015 01:28:35 GMT

I Agree. Sorry, using AH for the first time and accidentally clicked reply instead of comment 😞

Re: How to identify stale datanode?

amiller — Sun, 01 Nov 2015 07:54:30 GMT

No worries, we're all learning it as we go