host with secondary namenode failed, how do I instantiate on another host?

[#1]

Need to instantiate a secondary namenode on another host after hard drive failure

[#2]

@Wayne Hoyenga

You can register the new host to the cluster by using "Add Host" option in ambari and then once the host is registered to the ambari cluster then via ambari you can move the Standby NameNode from one host to another host as described in the following article:

https://community.hortonworks.com/articles/65837/migrate-namenode-from-one-machine-to-another-via-a....

.

[#3]

@Jay Kumar SenSharma

Step 1 of those instructions is "Ensure that all the HDFS Components are up and running". Given that the secondary namenode service cannot be brought up since the host cannot be booted, is there any chance the migration can be successful?

[#4]

@Wayne Hoyenga

Please ignore the Step1 and proceed with next steps. Ambari UI wizard will take care of the move of Standby Namenode.

Please note the "Move NameNode" operation from Ambari UI will perform the following operations:

 Stop Required Services
 Create NameNode+ZKFC
 Disable HDFS
 Reconfigure NameNode+ZKFC
 Install NameNode+ZKFC       (on new StandbyNN)
 Start ZooKeeper Servers
 Start NameNode

.

With some manual steps like:
Please be careful while running the below commands from the Correct Hosts as instructed by the Ambari UI.
Login to the NameNode host, Reset automatic failover information in ZooKeeper by running:

# sudo su hdfs -l -c 'hdfs zkfc -formatZK'

Login to the newly installed NameNode host. (Important!) Be sure to login to the newly installed NameNode host.
Initialize the metadata by running:

# sudo su hdfs -l -c 'hdfs namenode -bootstrapStandby'

.

[#5]

@Jay Kumar SenSharma

I've not had the best of luck with the move/migrate wizards. In this case, the wizard has stopped trying to shut down services. Since the wizard is running, I no longer have access to the Ambari UI.

I managed to get past that part of the wizard by using "/usr/hdp/current/oozie-server/bin/oozied.sh stop" to stop the Oozie service (that's what the wizard was hanging on).

The wizard completed and I then got to the manual steps.

The first step:

1. Copy the contents of /scratch/hadoop/hdfs/namesecondary,/hadoop/hdfs/namesecondary on the source host cg-hm10.ncsa.illinois.edu to /scratch/hadoop/hdfs/namesecondary,/hadoop/hdfs/namesecondary on the target host cg-hm16.ncsa.illinois.edu.

isn't possible as the directory on cg-hm10 no longer exists.