cloudera fails to resolve hosts

[#1]

Initially, i set up 2 machines (Ubuntu 12.04, x64) on vSphere server.

The name and ip of these two machines were

host                  ip
vm-cluster-node1      10.211.55.100
vm-cluster-node2      10.211.55.101

I have installed cloudera manager in vm-cluster-node1.

Then i cloned second one (vm-cluster-node2) to create 2 more hosts, and changed the ip and names as:

host                  ip
vm-cluster-node3      10.211.55.102
vm-cluster-node4      10.211.55.103

But the problem is, when i add these all 4 machines from cloudera, no matter how many times i try, i can only see two machines in hosts tab. later i realized that, if i refresh the web-page, i can see 2 machines only, but the second machine is switched between vm-cluster-node2, vm-cluster-node3 and vm-cluster-node4.

to illustrate, i have included images to make things clear.

So, as far i've understood, the cloudera manager is not able to resolve the hosts cloned from same source as different machines. Even though the host-names and IPs have been changed. So is there anything that is common in these machines and this problem is occurring?

[#2 · Accepted solution]

Thanks for the reply.

When i checked, i found the machines have different MAC addresses, but reverse DNS is not working.

However, i have found that the HOST_ID for the three hosts are same. Is this because i cloned the vms from same source. Could this be the reason for problem. If yes, how can i change the host-id of the hosts?

[#3]

The networking and security requirements for installation are listed here, make sure you have read through and configured based on this discussion.

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_cm_requirements.h...

each host must be able to perform forward and reverese lookup of the others.

if /etc/hosts is how you are managing dns, make sure they reflec the proper IP/Hostname

The other thing to verify is that your cloned vm's have unique MAC addresses for their network interfaces, you should be able to look at the physical (MAC) address on each node with the ifconfig command.

[#4 · Accepted solution]

Thanks for the reply.

When i checked, i found the machines have different MAC addresses, but reverse DNS is not working.

However, i have found that the HOST_ID for the three hosts are same. Is this because i cloned the vms from same source. Could this be the reason for problem. If yes, how can i change the host-id of the hosts?

[#5 · Accepted solution]

It could be that the hostID is what is causing the problem.  Changing the hostid value on your hosts will be something you need to referfence from your linux distribution's documentation.

I see references in our CM code where if we do not find a hostid, that the hostname is used instead, so having duplicate hostid's is probably creating issues.

Tood

[#6]

Yes that was the proble,

---

Tgrayson wrote:

It could be that the hostID is what is causing the problem.  Changing the hostid value on your hosts will be something you need to referfence from your linux distribution's documentation.

I see references in our CM code where if we do not find a hostid, that the hostname is used instead, so having duplicate hostid's is probably creating issues.

Tood

---

i did that.... i changed host id by modifingfollowing file:

/etc/hosts/cloudera-scm-agent 

by adding

CMF_AGENT_ARGS="--host_id new_host_id"

[#7]

Hi,

We have the same problem i.e. when we use 2 clones of the same VM and try to connect the Master node to them, only one of them is visible at a time. We have gone through the thread above and made sure that - 

i) Hostnames of the 2 VMs are different

ii) Hostids of the 2 VMs are different

iii) /etc/hosts has names and IPs of all 3 machines i.e. Master, Node1 and Node2

iv) Physical addresses of the 2 VMs is different

However, the problem persists. Any further inputs into what could be causing the issue please? Any help would be appreciated.

Thanks in advance.

Regards,

Yogesh

[#8]

Hi Yogesh,

Each agent will identify the host to Cloudera Manager with a "uuid" that is set by the agent.  So, if you have two hosts with the same uuid (which I assume could occur when cloning), Cloudera Manager would see them as the same host.

To verify this is the cause, look at the /var/lib/cloudera-scm-agent/uuid file on each host.

If the uuid value is the same for any hosts, find the hosts that DO NOT appear in Cloudera Manager and do this on the host that is not appearing in CM:

(1)

service cloudera-scm-agent stop

(2)

rm /var/lib/cloudera-scm-agent/uuid

(3)

service cloudera-scm-agent start

When the agent starts, it should generate a new unique uuid and when it heartbeats to Cloudera Manager, CM should now detect that it is a new host.

[#9]

This is fantastic Bgooley! Your solution worked. 

The cloned VM indeed had the same UUID as the parent. As per your instructions, we stopped the scm agent, deleted the uuid and restarted scm agent. We then added the node to the cluster via Cloudera Manager and it got registered correctly. We can now see both the VM nodes.

Thank you so much! Appreciate your help.