Nodes (running CM Agent) not all showing up on Cloudera Manager UI HOSTS section ...

[#1]

Hello Friends:

THE DETAILS

• I have CDH5.3 (latest) installed on five CentOS-6.6 nodes [vps00 - vps04]. The CDH packages were installed via traditional YUM repositories.
• Each node is actually a Linux LXC container with it's own IP address (192.168.0.[180-184]).
• Each node also has the (latest) ClM Agent packages installed and running; and each /etc/cloudera-scm-agent/config.ini points to the CM.
• The Cloudera Manager server itself, is also a CentOS-6.6 Linux LXC container (vps10), and it's IP address is 192.168.0.190.
• All six hosts can communicate with each other, without issue; and there are no port restrictions either.

We always run clusters via UNIX CLI without issue... As such HDFS, Map/Reduce jpbs, Storm, Zookeeper, and anything else you can think of, run

flawlessly.

THE PROBLEM:

We decided to try Cloudera Manager UI today, but not all vps hosts are showing up in the UI.

Although on vps10 (the Cloudera Manager server) we see agent connections to ALL vps nodes, the HOSTS section of the CM UI shows only one

vps node at a time; and refreshing the page changes which (single) vps is shown. Strange. It seems like every is configured correctly, too.

user@vps10$ sudo netstat -an | grep 192.168.0.18
tcp        0      0 192.168.0.190:7182 vps10    192.168.0.180:58605         ESTABLISHED vps00
tcp        0      0 192.168.0.190:7182 vps10    192.168.0.181:59878         ESTABLISHED vps01
tcp        0      0 192.168.0.190:7182 vps10    192.168.0.182:36202         ESTABLISHED vps02
tcp        0      0 192.168.0.190:7182 vps10    192.168.0.183:49203         ESTABLISHED vps03
tcp        0      0 192.168.0.190:7182 vps10    192.168.0.184:46649         ESTABLISHED vps04

Note that we want to simply add these (pre-configured) nodes manually to CM and to a Cluster within it, as they are already configured with CDH

and working.

Any ideas why all hosts aren't all showing up (... is only one showing up, and changes with each page refresh). Seems like a conflict. Hmm.

Thank you in advance,

PRISMAL

[#2]

Did you by chance configure all CM agent hosts with the same host id?
/var/lib/cloudera-scm-agent/uuid

[#3]

Hi DLO:

Thank you for the quick reply. Sadly, as it turns out, yes (good intuidion by you). =:)

After building a pristine LXC (vps00), I cloned it. And while I changed network information for each clone, I didn't change the UUID piece (as I wasn't aware of it). Thank you for bringing that to my attention. CONFIRMATION:

user@lxchost$ ssh -l user vps00 "cat /var/lib/cloudera-scm-agent/uuid"
e8b6ade3-7838-47ed-ba8e-99bd3e5f97b5
user@lxchost$ ssh -l user vps01 "cat /var/lib/cloudera-scm-agent/uuid"
e8b6ade3-7838-47ed-ba8e-99bd3e5f97b5
user@lxchost$ ssh -l user vps02 "cat /var/lib/cloudera-scm-agent/uuid"
e8b6ade3-7838-47ed-ba8e-99bd3e5f97b5
user@lxchost$ ssh -l user vps03 "cat /var/lib/cloudera-scm-agent/uuid"
e8b6ade3-7838-47ed-ba8e-99bd3e5f97b5
user@lxchost$ ssh -l user vps04 "cat /var/lib/cloudera-scm-agent/uuid"
e8b6ade3-7838-47ed-ba8e-99bd3e5f97b5

Can this be hand-edited manually to fix (after shutting down the agent first, of course)?

For example, changing the last two positions to make the UUIDs unique, like so?

> vps00  -- UUID would end in '...00'

> vps01  -- UUID would end in '...01'

> vps02  -- UUID would end in '...02'

> vps03  -- UUID would end in '...03'

> vps04  -- UUID would end in '...04'

Or, is there a preferred method for changing these? Also, do I have to purge some possibly

cached information on the CM server?

Finally, are/is there anything else lurking as non-unique in my clones (or just this UUID thing)?

=:)

Thank you again DLO,

PSIAMAL

[#4 · Accepted solution]

Hi Psiamal,

I believe you can just remove that file, then the CM agent will re-generate it with a unique ID for you.

I would stop all your agents, delete the hosts via the CM UI, delete the uuid files on the agent hosts, then start your agents. The hosts should appear correctly in the CM UI shortly.

I don't think there's any other things you need to purge from the clones. Can't say I've tested this myself though.

Thanks,
Darren

[#5]

Hi Darren:

I basically did what you prescribed, and it resolved the issue. I'll mark it as solved.

Thank you again,

prismal