Cloudera Manager stuck at Distributed on CentOS

[#1]

I've been trying to install Cloudera 5.10 on Centos 7.x in a VMWare environment.  Each time Cloudera Manager gets stuck at Distributed.  Any help appreciated.  Thanks. 

Screen shot

[#2]

The Distributed portion is the agents downloading it from the CM server. Check the CM Agent logs for clues. The obvious issues will be resolution of the CM server address or ports being blocked.

[#3]

The log files don't seem to indicate much.  I do see...

[05/Mar/2017 15:49:16 +0000] 13117 Thread-2 server       INFO     stats_alert CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel: [1004] 0 0 0 0 60 0 0 0 0 0
[05/Mar/2017 15:49:17 +0000] 13117 MainThread server       INFO     Adding torrent: file:///opt/cloudera/parcel-cache/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.torrent
[05/Mar/2017 15:49:17 +0000] 13117 MainThread rpc          ERROR    Failed handling RPC: AddTorrent Already present torrent: CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/flood/util/rpc.py", line 176, in executeRPC
    return fn(*args)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/flood/server.py", line 233, in addTorrent
    torrent = self.depot.importTorrent(torrentUrl)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/flood/depot.py", line 101, in importTorrent
    raise TorrentError("Already present torrent: %s", torrentName)
TorrentError: Already present torrent: CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel

The following messages also seem to be repeated forever...

[05/Mar/2017 17:03:35 +0000] 13117 Thread-2 server       INFO     stats_alert CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel: [1005] 0 0 0 0 60 0 0 0 0 0
[05/Mar/2017 17:03:36 +0000] 13117 Thread-2 server       INFO     stats_alert CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel: [1000] 0 0 0 0 60 0 0 0 0 0
[05/Mar/2017 17:03:37 +0000] 13117 Thread-2 server       INFO     stats_alert CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel: [1002] 0 0 0 0 60 0 0 0 0 0
[05/Mar/2017 17:03:38 +0000] 13117 Thread-2 server       INFO     stats_alert CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel: [1003] 0 0 0 0 60 0 0 0 0 0
[05/Mar/2017 17:03:39 +0000] 13117 Thread-2 server       INFO     stats_alert CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel: [1001] 0 0 0 0 60 0 0 0 0 0

[#4]

@Meister1867,

Based on the exception, it appears that distrobution started but got into an ambiguous state as the torrent file already existed.  I would recommend cleaning up to see if the agent can start the download again:

(1)

stop the agent with "service cloudera-scm-agent stop"

(2)

Delete all cached files from the "parcel-cache" which is the directory where torrent files are cached.

# rm -rf /opt/cloudera/parcel-cache/*

Delete all files that have been downloaded by the "flood" torrent mechanism

# rm -rf /opt/cloudera/parcels/.flood/*

The goal here is to clear out files that appear to be causing trouble for the download.  When we start the agent, it should detect it needs to start the download again.

(3)

Start the agent with "service cloudera-scm-agent start"

(4)

Monitor parcel status to see if this helps.

[#5]

I should add... I'm installing this on a single VM.  The idea is that I can then repeat these steps elsewhere.

After restarting the agent I see...

[08/Mar/2017 01:42:32 +0000] 44186 MainThread agent        INFO     CM server guid: c6d30813-4816-48d7-ba2f-7bdba5ea0d22
[08/Mar/2017 01:42:32 +0000] 44186 MainThread agent        INFO     Using parcels directory from server provided value: /opt/cloudera/parcels
[08/Mar/2017 01:42:32 +0000] 44186 MainThread parcel       INFO     Agent does create users/groups and apply file permissions
[08/Mar/2017 01:42:32 +0000] 44186 MainThread parcel_cache INFO     Using /opt/cloudera/parcel-cache for parcel cache
[08/Mar/2017 01:42:32 +0000] 44186 MainThread agent        ERROR    Caught unexpected exception in main loop.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/agent.py", line 710, in __issue_heartbeat
    self._init_after_first_heartbeat_response(resp_data)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/agent.py", line 947, in _init_after_first_heartbeat_response
    self.client_configs.load()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/client_configs.py", line 682, in load
    new_deployed.update(self._lookup_alternatives(fname))
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/client_configs.py", line 432, in _lookup_alternatives
    return self._parse_alternatives(alt_name, out)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/client_configs.py", line 444, in _parse_alternatives
    path, _, _, priority_str = line.rstrip().split(" ")
ValueError: too many values to unpack

I'm not sure why I'm getting so many issues. 

[#6 · Accepted solution]

Solved distribution issue by adding the VM's IP and FQDN to /etc/hosts

[#7]

Thanks 🙂