Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

One NameNode of NameNode HA goes down and reinstall CentOS and then join cluster again

Highlighted

One NameNode of NameNode HA goes down and reinstall CentOS and then join cluster again

New Contributor

Hi all,

We have install a Hadoop cluster with NameNode HA and HBase HA, that say NameNode and HMaster are on the same node, and we have two nodes to do the HA. Just now, there is an problem happen on one node, so we reinstall the operating system on that node. Now we want to join that node into the Hadoop cluster. But we do not know how to do it. Please help us and thanks in advances!

7 REPLIES 7

Re: One NameNode of NameNode HA goes down and reinstall CentOS and then join cluster again

Super Guru
@xu jerry

1. Install ambari-agent and do manual registration (i.e. put hostname/ip-address of ambari server in /etc/ambari-agent/conf/ambari-agent.ini

2. Now it should be part of your cluster and you can install missing services again from Ambari UI.

Re: One NameNode of NameNode HA goes down and reinstall CentOS and then join cluster again

Expert Contributor

@xu jerry

You need to treat this node as new node and installation process would be similar as Adding new node.

1. Install Ambari-repo on this node.

2. Install ambari-agent package

3. Edit /etc/ambari-agent/conf/ambari-agent.ini and update hostname value under [server] block. Supply ambari-server hostname here.

4. Start ambari-agent service

5. Open browser with URL : http://<AMBARI-SERVER-IP/FQN>:8080/api/v1/hosts

6. If you see FQDN of new node in above webpage , this means ambari-agent is now able to communicate with ambari-server.

7. Go to ambari-server and Add this node and install required services. This page shows how to install services on new host after adding to cluster. : http://hortonworks.com/hadoop-tutorial/using-apache-ambari-add-new-nodes-existing-cluster/

Hope this helps.

Re: One NameNode of NameNode HA goes down and reinstall CentOS and then join cluster again

New Contributor

In addition to what Pradeep said, the way to add new "master nodes" is to add new "worker nodes", then to assign master services to that node. Have a look at this https://community.hortonworks.com/answers/31935/view.html too.

Re: One NameNode of NameNode HA goes down and reinstall CentOS and then join cluster again

New Contributor

Thank u very much!

@Kuldeep Kulkarni, Do you mean I can set the down node with the same IP it has before been reinstall operating system? And I just use ambari to reinstall the missing services via Ambari UI?

@Pradeep Bhadani

Can I set the down node with the same IP it has before been reinstall operating system? Or I have to use another IP and treat it as a exactly new node?

Re: One NameNode of NameNode HA goes down and reinstall CentOS and then join cluster again

Super Guru

@xu jerry - Yes. If IP is different and hostname is same then it should be fine as Ambari keeps only hostname information in its DB.

Before re-installing you need to install ambari-agent on formatted host followed by manual registration then it should become part of your cluster and you can install services as per your requirements.

Happy Hadooping! :)

Re: One NameNode of NameNode HA goes down and reinstall CentOS and then join cluster again

New Contributor

By the way, now I simulate such scene:

1. reinstall ambari-agent on the fail node

2. set the ambari-agent.ini

3. run 'yum remove hbase' to uninstall the hbase service

4. run 'service ambari-agent restart' to start the ambari-agent

5. go to ambari UI to reinstall hbase : but I just find there are two options: "Start" and "Turn On Maintenance Mode", so I fail to reinstall the hbase. I think there should be a flag in the ambari database which records whether or not the node has installed hbase successfully. So if I can change the flag to be false I can restall HBase.

Re: One NameNode of NameNode HA goes down and reinstall CentOS and then join cluster again

Expert Contributor

@xu jerry How did you removed node from cluster - decomissioned node or deleted node from Ambari before reinstall OS?

Don't have an account?
Coming from Hortonworks? Activate your account here