question Re: HA Fails when Namenode server rebooted in Support Questions

HA Fails when Namenode server rebooted

shrikant_bm — Sun, 05 Jul 2020 18:19:16 GMT

HI Team,

I came across an scenario where HA failed when active namenode server was rebooted.

Accidentally the master node which runs active namenode got rebooted.

when it got rebooted i believed the standby namenode will become active namenode and will continue the operations but standby namenode did not became active and HDFS was completely down.

So my question is

1. When active namenode server is rebooted will the standby namenode will not become active? Is this something expected? Or Did the HA did not work in our cluster

2. Whether the HA is expected to work only between the active and standby namenode daemons?

Thanking in Advance!!

Re: HA Fails when Namenode server rebooted

Madhur — Mon, 06 Jul 2020 07:42:14 GMT

Hi @shrikant_bm ,

Thank you for reaching out to community!

If NameNode high availability is enabled and configured on your cluster. Automatic failover of active NameNode should work.

[1] Give us steps on configuring NN high availability using Ambari.

https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/managing-high-availability/content/amb_enable_namenode_high_availability.html

[2] Gives us steps on Managing High Availability of Services for other components.

https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/managing-high-availability/content/amb_managing_high_availability_of_services.html

Re: HA Fails when Namenode server rebooted

shrikant_bm — Mon, 06 Jul 2020 09:16:20 GMT

@Madhur : Thanks for the update!!

Thanks for sharing links.

But I am looking for answers for my below questions

1. When active namenode server is rebooted will the standby namenode will not become active? Is this something expected? Or Did the HA did not work in our cluster

2. Whether the HA is expected to work only between the active and standby namenode daemons?

Re: HA Fails when Namenode server rebooted

Madhur — Mon, 06 Jul 2020 10:41:09 GMT

Hi @shrikant_bm ,

Please find answers inline.

1. When active namenode server is rebooted will the standby namenode will not become active? Is this something expected? Or Did the HA did not work in our cluster

Yes, standby NameNode will become active when primary NameNode reboots, provided high availability is enabled and configured.

2. Whether the HA is expected to work only between the active and standby namenode daemons?

Yes, HA works only between active and standby NameNode. It is taken care by ZKFC (ZooKeeper Failover Controller).

Re: HA Fails when Namenode server rebooted

shrikant_bm — Tue, 07 Jul 2020 11:06:57 GMT

@Madhur Thanks for the update.

So I can conclude that HA must work in both the below 2 cases

1. HA must work whenever active namenode daemon goes down.

2. HA must work whenever active namenode server goes down.

Please note that i am mentioning two thing i,e active namenode daemon and active namenode server

Re: HA Fails when Namenode server rebooted

Madhur — Tue, 07 Jul 2020 12:05:07 GMT

Hi @shrikant_bm ,

When ever Active NameNode server goes down, its associated daemon also goes down. HA works in the same way whenever Active NameNode daemon or server goes down. ZKFC will not receive the heartbeat and the ZooKeeper session will expire, notifying the other NameNode that a failover should be triggered.

To answer your question: yes, in both the cases mentioned by you, HA should work.