Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Failed to connect to previous supervisor

SOLVED Go to solution

Failed to connect to previous supervisor

Explorer

I've chosen to install CDH through Automated installer using Cloudera Manager, the download completes but unable to push through due this error:

 

Installation failed. Failed to receive heartbeat from agent.
Ensure that the host's hostname is configured properly.
Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
Ensure that ports 9000 and 9001 are not in use on the host being added.
Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).
If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.

However upon checking the details, I saw Failed to connect to previous supervisor in this error details:

Installation script completed successfully.
all done
closing logging file descriptor
>>[18/Jul/2017 01:34:30 +0000] 4355 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor
>>[18/Jul/2017 01:34:30 +0000] 4355 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/flood
>>[18/Jul/2017 01:34:30 +0000] 4355 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include
>>[18/Jul/2017 01:34:30 +0000] 4355 MainThread agent ERROR Failed to connect to previous supervisor.
>>Traceback (most recent call last):
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2109, in find_or_start_supervisor
>> self.configure_supervisor_clients()
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2290, in configure_supervisor_clients
>> supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")])
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 1599, in realize
>> Options.realize(self, *arg, **kw)
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 333, in realize
>> self.process_config()
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 341, in process_config
>> self.process_config_file(do_usage)
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 376, in process_config_file
>> self.usage(str(msg))
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 164, in usage
>> self.exit(2)
>>SystemExit: 2
>>[18/Jul/2017 01:34:30 +0000] 4355 Dummy-1 daemonize WARNING Stopping daemon.
>>[18/Jul/2017 01:34:30 +0000] 4355 Dummy-1 agent INFO Stopping agent...
>>[18/Jul/2017 01:34:30 +0000] 4355 Dummy-1 agent INFO No extant cgroups; unmounting any cgroup roots
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO SCM Agent Version: 5.12.0
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO Agent Protocol Version: 4
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO Using Host ID: b9e306ab-b527-4667-9f3e-b6acad9f5224
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO Using directory: /run/cloudera-scm-agent
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO Using supervisor binary path: /usr/lib64/cmf/agent/build/env/bin/supervisord
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation.
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO Agent Logging Level: INFO
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO No command line vars
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO Missing database jar: /usr/share/java/mysql-connector-java.jar (normal, if you're not using this database type)
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type)
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
>>[18/Jul/2017 01:39:14 +0000] 5611 MainThread agent INFO Agent starting as pid 5611 user root(0) group root(0).


This is my current setup.

CentOS 7.2
Installing CDH 5.11.1 or 5.12 using Cloudera Manager.
4 nodes

/etc/hosts

192.168.0.101 node1.cirro.com node1
192.168.0.102 node2.cirro.com node2
192.168.0.103 node3.cirro.com node3
192.168.0.104 node4.cirro.com node4

/etc/sysconfig/network

NETWORKING=yes
HOSTNAME=myservers*.cirro.com
NOZEROCONF=yes

/etc/ssh/sshd_config

PermitRootLogin yes
PasswordAuthentication yes

hostname has also been set per node to reflect /etc/sysconfig/network.

sestatus = disabled
firewalld = inactive
ntpd = active (running)
httpd = active (running)
vm.swappiness = 10
user = passwordless sudo
/etc/rc.local has been set

Can anyone help me on this? I've been stuck with this for 2 weeks now. I've run out of options and searching online. It would be really appreciated!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Failed to connect to previous supervisor

Explorer

This is what I did instead. I followed Path B and downloaded 5.11.1 version instead. Solved all of my problems.

29 REPLIES 29

Re: Failed to connect to previous supervisor

New Contributor

hello,

 

anyone can help ?  I have same issue. Tried on centos and ubuntu. firewall & dns & hosts are totally OK. 

Re: Failed to connect to previous supervisor

Champion

Whats the status of cloudera-scm-server,Cloudera-scm-server-db,Cloudera-scm-agent ? 

I assume you are using Path A  - installer.bin script ? right 

 

did you try loggin in cloudera manager web ui like http://localhost:7180 or http://your_hostname:7180.

whats ur kernel version you are ruining ? 

Re: Failed to connect to previous supervisor

New Contributor

uname -a
Linux hadoop-uat201 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

 

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty

 

 

/etc/init.d/cloudera-scm-server status
Checking for service cloudera-scm-server: * cloudera-scm-server is running

 

/etc/init.d/cloudera-scm-server-db status
* Cloudera manager database is running

 

/etc/init.d/cloudera-scm-agent status
Checking for service cloudera-scm-agent * cloudera-scm-agent is not running  - It fails on start because of supevisor daemon. It is unaccessible. and it is not started during installation without any error. i am thinking about move supervisord.conf from working cluster in order to start supervisord 

 

Re: Failed to connect to previous supervisor

New Contributor

i am really believe that something wrong with cloudera-manager-installer.bin package.  I have already installed 10 clusters of different CM versions and can do it with closed eye. But i am stuck with this new version.

Re: Failed to connect to previous supervisor

Explorer

Believe it or not, I made it work using 5.11.1 in my first try, then came the 5.12, I could no longer install any package. There must be something wrong with the new release.

Re: Failed to connect to previous supervisor

New Contributor

and another strange thing. I am trying to install old versions (which i have installed many times ) and download .bin packege directly for 11 version from repository (for example wget http://archive.cloudera.com/cm5/installer/5.11.1/cloudera-manager-installer.bin).  But the only option avaliable during installation is 5.12 ! Tried a few times with the same results...

Re: Failed to connect to previous supervisor

Champion

@darkdante

 

The commuity had some kernel version compatability issue a month back if I remember 

You should be  good with 5.12 we are using 5.12 guess what HUE interface has some new functionalties and many more 

Re: Failed to connect to previous supervisor

New Contributor

I've experiened the same problem on Ubuntu 16.04 LTS with Cloudera CDH 5.12

 

Out of 5 nodes, 4 upgraded with no problems, while the 5th has mentioned issue. All instances were working fine on CDH 5.11

 

One clue is the fact that supervisord directories were created but are empty

 

Is there a way to enable additional logging for supervisord during installation?

Re: Failed to connect to previous supervisor

Champion

see if you have multiple supervisord runining 

ps aux   |   grep supervisor

share the logs  to see whats it is complain about supervisord