Skip to main content
KPG1
KPG1
Gold ⭐️
July 18, 2022
Solved

frequently getting stale alerts for Datanodes.

Hi everyone.

I am getting stale alerts every hr, if the last point of contact between Data nodes and Namenode is more than 30 s we get these alerts.

I am not able to find the root cause of this slowness, I have 32 cores system, but when this alert is generated in htop hdfs usage is more but not all cores are 100% utilized.

DataNode Health Summary

DataNode Health: [Live=5, Stale=1, Dead=0]
Please suggest changes required to resolve this.
rki_
rki_Answer
Gold ⭐️
July 18, 2022

Hello @KPG1 ,

 

The time taken to mark a datanode as stale is give by dfs.namenode.stale.datanode.interval, with a default of 30 seconds. If this is happening with a specific Datanode, you can check if there is any network issues between the Datanode and the Namenode or if the Datanode has any JVM pauses reported by checking the Datanode logs. As a bandaid, you can bump up the above parameter till the underlying problem is solved.

View in thread

1 reply

rki_
Gold ⭐️
rki_Answer
Gold ⭐️
July 18, 2022

Hello @KPG1 ,

 

The time taken to mark a datanode as stale is give by dfs.namenode.stale.datanode.interval, with a default of 30 seconds. If this is happening with a specific Datanode, you can check if there is any network issues between the Datanode and the Namenode or if the Datanode has any JVM pauses reported by checking the Datanode logs. As a bandaid, you can bump up the above parameter till the underlying problem is solved.

Translate
    Terms & ConditionsAccessibility statement