question frequently getting stale alerts for Datanodes. in Support Questions

frequently getting stale alerts for Datanodes.

KPG1 — Mon, 18 Jul 2022 04:48:08 GMT

Hi everyone.

I am getting stale alerts every hr, if the last point of contact between Data nodes and Namenode is more than 30 s we get these alerts.

I am not able to find the root cause of this slowness, I have 32 cores system, but when this alert is generated in htop hdfs usage is more but not all cores are 100% utilized.

DataNode Health Summary

DataNode Health: [Live=5, Stale=1, Dead=0]

Please suggest changes required to resolve this.

Re: frequently getting stale alerts for Datanodes.

rki_ — Mon, 18 Jul 2022 09:00:19 GMT

Hello @KPG1 ,

The time taken to mark a datanode as stale is give by dfs.namenode.stale.datanode.interval, with a default of 30 seconds. If this is happening with a specific Datanode, you can check if there is any network issues between the Datanode and the Namenode or if the Datanode has any JVM pauses reported by checking the Datanode logs. As a bandaid, you can bump up the above parameter till the underlying problem is solved.