HA Fails when Namenode server rebooted

[#1]

HI Team,

I came across an scenario where HA failed when active namenode server was rebooted.

Accidentally the master node which runs active namenode got rebooted.

when it got rebooted i believed the standby namenode will become active namenode and will continue the operations but standby namenode did not became active and HDFS was completely down.

So my question is

1. When active namenode server is rebooted will the standby namenode will not become active? Is this something expected? Or Did the HA did not work in our cluster

2. Whether the HA is expected to work only between the active and standby namenode daemons?

Thanking in Advance!!

[#2]

Hi @shrikant_bm ,

Thank you for reaching out to community! 

If NameNode high availability is enabled and configured on your cluster. Automatic failover of active NameNode should work.

[1] Give us steps on configuring NN high availability using Ambari.

https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/managing-high-availability/content/amb_enable_...

[2] Gives us steps on Managing High Availability of Services for other components.

https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/managing-high-availability/content/amb_managin...

[#3]

@Madhur : Thanks for the update!!

Thanks for sharing links.

But I am looking for answers for my below questions

1. When active namenode server is rebooted will the standby namenode will not become active? Is this something expected? Or Did the HA did not work in our cluster

2. Whether the HA is expected to work only between the active and standby namenode daemons?

[#4]

Hi @shrikant_bm ,

Please find answers inline.

1. When active namenode server is rebooted will the standby namenode will not become active? Is this something expected? Or Did the HA did not work in our cluster

Yes, standby NameNode will become active when primary NameNode reboots, provided high availability is enabled and configured. 

2. Whether the HA is expected to work only between the active and standby namenode daemons?

Yes, HA works only between active and standby NameNode. It is taken care by ZKFC (ZooKeeper Failover Controller).

[#5]

@Madhur Thanks for the update.

So I can conclude that HA must work in both the below 2 cases

1. HA must work whenever active namenode daemon goes down.

2. HA must work whenever active namenode server goes down.

Please note that i am mentioning two thing i,e active namenode daemon and active namenode server

[#6 · Accepted solution]

Hi @shrikant_bm ,

When ever Active NameNode server goes down, its associated daemon also goes down. HA works in the same way whenever Active NameNode daemon or server goes down. ZKFC will not receive the heartbeat and the ZooKeeper session will expire, notifying the other NameNode that a failover should be triggered.

To answer your question: yes, in both the cases mentioned by you, HA should work.