HDFS_CANARY_HEALTH Failure and "Cannot create a file. Namenode in safe mode" in NameNode HA

[#1]

I have seen a situation wher some of the jobs suddenly failed with error saying "Cannot create a file. Namenode in safe mode." When I check the cloudera manager, HDFS_CANARY_HEALTH check failed. I also observed that the Active namenode become standaby and standby become active. So, if the HDFS_CANARY_HEALTH test fails then Namenode restarts by itself? Can anyone respond to this.

[#2 · Accepted solution]

The canary test does not have anything with the failover. This just reports to the Cloudera Manager the health status. The actual failover in HA scenario is initiated by a Failover controller(s) - you have probably two NN, 3 (or 5) JNodes and Failover controllers. When the Active NN goes down, the FC brings the standby NN into a Active mode