One NameNode of NameNode HA goes down and reinstall CentOS and then join cluster again

[#1]

Hi all,

We have install a Hadoop cluster with NameNode HA and HBase HA, that say NameNode and HMaster are on the same node, and we have two nodes to do the HA. Just now, there is an problem happen on one node, so we reinstall the operating system on that node. Now we want to join that node into the Hadoop cluster. But we do not know how to do it. Please help us and thanks in advances!

[#2]

@xu jerry

1. Install ambari-agent and do manual registration (i.e. put hostname/ip-address of ambari server in /etc/ambari-agent/conf/ambari-agent.ini

2. Now it should be part of your cluster and you can install missing services again from Ambari UI.

[#3]

@xu jerry

You need to treat this node as new node and installation process would be similar as Adding new node.

1. Install Ambari-repo on this node.

2. Install ambari-agent package

3. Edit /etc/ambari-agent/conf/ambari-agent.ini and update hostname value under [server] block. Supply ambari-server hostname here.

4. Start ambari-agent service

5. Open browser with URL : http://<AMBARI-SERVER-IP/FQN>:8080/api/v1/hosts

6. If you see FQDN of new node in above webpage , this means ambari-agent is now able to communicate with ambari-server.

7. Go to ambari-server and Add this node and install required services. This page shows how to install services on new host after adding to cluster. : http://hortonworks.com/hadoop-tutorial/using-apache-ambari-add-new-nodes-existing-cluster/

Hope this helps.

[#4]

In addition to what Pradeep said, the way to add new "master nodes" is to add new "worker nodes", then to assign master services to that node. Have a look at this https://community.hortonworks.com/answers/31935/view.html too.

[#5]

Thank u very much!

@Kuldeep Kulkarni, Do you mean I can set the down node with the same IP it has before been reinstall operating system? And I just use ambari to reinstall the missing services via Ambari UI?

@Pradeep Bhadani

Can I set the down node with the same IP it has before been reinstall operating system? Or I have to use another IP and treat it as a exactly new node?

[#6]

@xu jerry - Yes. If IP is different and hostname is same then it should be fine as Ambari keeps only hostname information in its DB.

Before re-installing you need to install ambari-agent on formatted host followed by manual registration then it should become part of your cluster and you can install services as per your requirements.

Happy Hadooping! :)

[#7]

By the way, now I simulate such scene:

1. reinstall ambari-agent on the fail node

2. set the ambari-agent.ini

3. run 'yum remove hbase' to uninstall the hbase service

4. run 'service ambari-agent restart' to start the ambari-agent

5. go to ambari UI to reinstall hbase : but I just find there are two options: "Start" and "Turn On Maintenance Mode", so I fail to reinstall the hbase. I think there should be a flag in the ambari database which records whether or not the node has installed hbase successfully. So if I can change the flag to be false I can restall HBase.

[#8]

@xu jerry How did you removed node from cluster - decomissioned node or deleted node from Ambari before reinstall OS?