question Re: How to do a cluster failover from active namenode to standby namenode from Ambari console? in Archives of Support Questions (Read Only)

How to do a cluster failover from active namenode to standby namenode from Ambari console?

muthukumar_siva — Thu, 28 Jul 2016 16:19:54 GMT

How to do a cluster failover from active namenode to standby namenode from Ambari console?

Could someone please help with the exact steps to be done?

Re: How to do a cluster failover from active namenode to standby namenode from Ambari console?

sbhat — Thu, 28 Jul 2016 16:30:32 GMT

@Muthukumar S The cluster failover is automatically done by ambari. If you want to make your standby namenode the active one, you can stop the current active namenode (ensuring the current standby namenode is alive) and the current standby namenode will take over as the active namenode.

To manually change it check : https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html#Administrative_commands

Re: How to do a cluster failover from active namenode to standby namenode from Ambari console?

muthukumar_siva — Thu, 28 Jul 2016 16:30:54 GMT

@sbhat Thank you will try and let you know, once i bring down active name node service, standby which is active will become active namenode. After that if I start the name node service on the one where is stopped will be become standby? Also do I need to do any manual steps like making active name node is safe mode and run savenamespace and then stop the service? or putting into maintenance mode the active name node will take care of this?

Re: How to do a cluster failover from active namenode to standby namenode from Ambari console?

sbhat — Thu, 28 Jul 2016 16:38:22 GMT

@Muthukumar S Yes. That is correct. You will not have to do all of that. HDFS is built to be fault tolerant. So it should work seemlessly. But I would still prefer the second method if it is a production box. Do update your findings and upvote the answer if it works. Thanks!

Re: How to do a cluster failover from active namenode to standby namenode from Ambari console?

sshimpi — Thu, 28 Jul 2016 17:57:17 GMT

@Muthukumar S

In addition to @sbhat yes ambari will take care of failover, but if you intend to do the failover manually using cli the you can check for below command -

[hdfs@test ~]$ hdfs haadmin

Usage: haadmin

[-transitionToActive [--forceactive] <serviceId>]

[-transitionToStandby <serviceId>]

[-failover [--forcefence] [--forceactive] <serviceId> <serviceId>]

[-getServiceState <serviceId>]

[-checkHealth <serviceId>]

[-help <command>]

Re: How to do a cluster failover from active namenode to standby namenode from Ambari console?

ArpitAgarwal — Tue, 02 Aug 2016 03:16:44 GMT

If you have setup automatic failover with ZooKeeper Failover Controllers then the ZKFC processes will automatically transition the Standby NN to Active status if the current active is unresponsive. The decision about which NN should be made active is taken by the ZKFC instances (coordinating via ZooKeeper). Ambari does not decide which NN should be active.

If you wish to perform a manual failover then you can use the hdfs dfsadmin command as @Sagar Shimpi suggested.

Both alternatives are described in the HDP documentation:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_hadoop-ha/content/ha-nn-deploy-nn-cluster.html

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_hadoop-ha/content/nn-ha-auto-failover.html

If you want to better understand the internals of automatic NN failover (recommended if you are administering a Hadoop cluster with HA), I recommend reading the Apache docs, specifically the section on Automatic Failover.

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html