Created 11-03-2017 01:54 PM
Need to instantiate a secondary namenode on another host after hard drive failure
Created 11-03-2017 02:00 PM
You can register the new host to the cluster by using "Add Host" option in ambari and then once the host is registered to the ambari cluster then via ambari you can move the Standby NameNode from one host to another host as described in the following article:
Created 11-03-2017 02:10 PM
Step 1 of those instructions is "Ensure that all the HDFS Components are up and running". Given that the secondary namenode service cannot be brought up since the host cannot be booted, is there any chance the migration can be successful?
Created 11-03-2017 02:32 PM
Please ignore the Step1 and proceed with next steps. Ambari UI wizard will take care of the move of Standby Namenode.
Please note the "Move NameNode" operation from Ambari UI will perform the following operations:
Stop Required Services Create NameNode+ZKFC Disable HDFS Reconfigure NameNode+ZKFC Install NameNode+ZKFC (on new StandbyNN) Start ZooKeeper Servers Start NameNode
With some manual steps like:
Please be careful while running the below commands from the Correct Hosts as instructed by the Ambari UI.
Login to the NameNode host, Reset automatic failover information in ZooKeeper by running:
# sudo su hdfs -l -c 'hdfs zkfc -formatZK'
Login to the newly installed NameNode host. (Important!) Be sure to login to the newly installed NameNode host.
Initialize the metadata by running:
# sudo su hdfs -l -c 'hdfs namenode -bootstrapStandby'
Created 11-03-2017 03:58 PM
I've not had the best of luck with the move/migrate wizards. In this case, the wizard has stopped trying to shut down services. Since the wizard is running, I no longer have access to the Ambari UI.
I managed to get past that part of the wizard by using "/usr/hdp/current/oozie-server/bin/oozied.sh stop" to stop the Oozie service (that's what the wizard was hanging on).
The wizard completed and I then got to the manual steps.
The first step:
isn't possible as the directory on cg-hm10 no longer exists.