Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Installation failed. Failed to receive heartbeat. Configured hostname, DNS and FQDN correctly.

Installation failed. Failed to receive heartbeat. Configured hostname, DNS and FQDN correctly.

New Contributor

I have followed Installation path A and I also have configured hostname, FQDN, IP correctly as mentioned in another threads. I still face "failed to recieve heartbeat" error while installing cloudera manager.

Installation failed. Failed to receive heartbeat from agent.

    Ensure that the host's hostname is configured properly.
    Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
    Ensure that ports 9000 and 9001 are not in use on the host being added.
    Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).
    If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.

I have checked socket stats and 9000 and 9001 are not in use. Host name is configured properly and resolvable. I have cross-checked it using nslookup. I am not using TLS Encyption.

 

I am attaching cloudera-scm-logs.

 

  Following is /var/log/cloudera-scm-agent/cloudera-scm-agent.log

3634 [16/May/2017 13:27:08 +0000] 11630 MainThread agent        INFO     Trying to connect to newly launched supervisor (Attempt 5)
3635 [16/May/2017 13:27:08 +0000] 11630 MainThread agent        ERROR    Failed! trying again in 1 second(s)
3636 Traceback (most recent call last):
3637   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.0-py2.7.egg/cmf/agent.py", line 2192, in connect_to_new_supervis     or
3638     self.get_supervisor_process_info()
3639   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.0-py2.7.egg/cmf/agent.py", line 2212, in get_supervisor_process_     info
3640     self.identifier = self.supervisor_client.supervisor.getIdentification()
3641   File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__
3642     return self.__send(self.__name, args)
3643   File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request
3644     verbose=self.__verbose
3645   File "/usr/lib/python2.7/dist-packages/supervisor/xmlrpc.py", line 460, in request
3646     self.connection.request('POST', handler, request_body, self.headers)
3647   File "/usr/lib/python2.7/httplib.py", line 1017, in request
3648     self._send_request(method, url, body, headers)
3649   File "/usr/lib/python2.7/httplib.py", line 1051, in _send_request
3650     self.endheaders(body)
3651   File "/usr/lib/python2.7/httplib.py", line 1013, in endheaders
3652     self._send_output(message_body) 
3653   File "/usr/lib/python2.7/httplib.py", line 864, in _send_output
3654     self.send(msg)
3655   File "/usr/lib/python2.7/httplib.py", line 826, in send
3656     self.connect()
3657   File "/usr/lib/python2.7/httplib.py", line 807, in connect
3658     self.timeout, self.source_address)
3659   File "/usr/lib/python2.7/socket.py", line 571, in create_connection
3660     raise err
3661 error: [Errno 111] Connection refused
3662 [16/May/2017 13:27:08 +0000] 11630 MainThread agent        ERROR    Failed to connect to newly launched supervisor. Agent will exit
3663 [16/May/2017 13:27:08 +0000] 11630 MainThread agent        INFO     Stopping agent...
3664 [16/May/2017 13:27:08 +0000] 11630 MainThread agent        INFO     No extant cgroups; unmounting any cgroup roots
3665 [16/May/2017 13:27:08 +0000] 11630 Dummy-1 daemonize    WARNING  Stopping daemon.

Following is /var/log/cloudera-scm-agent/cloudera-scm-agent.out

 

130 [16/May/2017 10:47:17 +0000] 2924 MainThread agent        WARNING  Expected mode 0751 for /run/cloudera-scm-agent but was 0755
131 [16/May/2017 10:47:17 +0000] 2924 MainThread agent        INFO     Re-using pre-existing directory: /run/cloudera-scm-agent
132 [16/May/2017 12:24:51 +0000] 9818 MainThread agent        INFO     SCM Agent Version: 5.11.0
133 [16/May/2017 12:24:51 +0000] 9818 MainThread agent        WARNING  Expected mode 0751 for /run/cloudera-scm-agent but was 0755
134 [16/May/2017 12:24:51 +0000] 9818 MainThread agent        INFO     Re-using pre-existing directory: /run/cloudera-scm-agent
135 [16/May/2017 12:41:31 +0000] 10575 MainThread agent        INFO     SCM Agent Version: 5.11.0
136 [16/May/2017 12:41:31 +0000] 10575 MainThread agent        WARNING  Expected mode 0751 for /run/cloudera-scm-agent but was 0755
137 [16/May/2017 12:41:31 +0000] 10575 MainThread agent        INFO     Re-using pre-existing directory: /run/cloudera-scm-agent
138 [16/May/2017 13:27:01 +0000] 11616 MainThread agent        INFO     SCM Agent Version: 5.11.0
139 [16/May/2017 13:27:01 +0000] 11616 MainThread agent        WARNING  Expected mode 0751 for /run/cloudera-scm-agent but was 0755
140 [16/May/2017 13:27:01 +0000] 11616 MainThread agent        INFO     Re-using pre-existing directory: /run/cloudera-scm-agent

Following is /var/log/cloudera-scm-agent/supervisord.log

 

  1 2017-05-15 22:04:54,918 CRIT Supervisor running as root (no user in config file)
  2 2017-05-15 22:04:54,918 WARN Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
  3 2017-05-15 22:04:54,935 INFO RPC interface 'supervisor' initialized
  4 2017-05-15 22:04:54,935 INFO RPC interface 'supervisor' initialized
  5 2017-05-15 22:04:54,936 INFO daemonizing the supervisord process
  6 2017-05-15 22:04:54,936 INFO supervisord started with pid 7699
  7 2017-05-15 22:04:55,938 INFO spawned: 'cmflistener' with pid 7702
  8 2017-05-15 22:04:57,054 INFO success: cmflistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
  9 2017-05-15 22:05:03,469 WARN received SIGTERM indicating exit request
 10 2017-05-15 22:05:03,469 INFO waiting for cmflistener to die
 11 2017-05-15 22:05:03,470 INFO stopped: cmflistener (terminated by SIGTERM)
 12 2017-05-15 22:06:33,258 CRIT Supervisor running as root (no user in config file)
 13 2017-05-15 22:06:33,258 WARN Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
 14 2017-05-15 22:06:33,281 INFO RPC interface 'supervisor' initialized
 15 2017-05-15 22:06:33,282 INFO RPC interface 'supervisor' initialized
 16 2017-05-15 22:06:33,282 INFO daemonizing the supervisord process
 17 2017-05-15 22:06:33,282 INFO supervisord started with pid 7866
 18 2017-05-15 22:06:34,285 INFO spawned: 'cmflistener' with pid 7874
 19 2017-05-15 22:06:35,394 INFO success: cmflistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 20 2017-05-15 22:10:18,744 WARN received SIGTERM indicating exit request
 21 2017-05-15 22:10:18,744 INFO waiting for cmflistener to die
 22 2017-05-15 22:10:18,745 INFO stopped: cmflistener (terminated by SIGTERM)

Following is /var/log/cloudera-scm-agent/supervisord.out

 

  1 Traceback (most recent call last):
  2   File "/usr/lib/cmf/agent/build/env/bin/supervisord", line 12, in <module>
  3     load_entry_point('supervisor==3.0', 'console_scripts', 'supervisord')()
  4   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/supervisord.py", line 372, in main
  5     go(options)
  6   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/supervisord.py", line 382, in go
  7     d.main()
  8   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/supervisord.py", line 89, in main
  9     info_messages)
 10   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 1414, in make_logger
 11     stdout = self.nodaemon,
 12   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/loggers.py", line 344, in getLogger
 13     handlers.append(RotatingFileHandler(filename,'a',maxbytes,backups))
 14   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/loggers.py", line 172, in __init__
 15     FileHandler.__init__(self, filename, mode)
 16   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/loggers.py", line 98, in __init__
 17     self.stream = open(filename, mode)
 18 IOError: [Errno 13] Permission denied: '/var/log/cloudera-scm-agent/supervisord.log'

Following is /var/log/cloudera-scm-agent/cmf_listener.log

 

  1 [12/May/2017 16:55:49 +0000] 45082 MainThread supervisor_listener INFO     Starting event listener as pid 45082
  2 [12/May/2017 16:55:50 +0000] 45082 MainThread supervisor_listener INFO     Opened agent FIFO
  3 [12/May/2017 18:55:38 +0000] 66466 MainThread supervisor_listener INFO     Starting event listener as pid 66466
  4 [12/May/2017 18:55:39 +0000] 66466 MainThread supervisor_listener INFO     Cannot open agent FIFO (agent probably dead), dropping event
  5 [15/May/2017 21:02:42 +0000] 6293 MainThread supervisor_listener INFO     Starting event listener as pid 6293
  6 [15/May/2017 21:02:43 +0000] 6293 MainThread supervisor_listener INFO     Cannot open agent FIFO (agent probably dead), dropping event
  7 [15/May/2017 22:04:56 +0000] 7702 MainThread supervisor_listener INFO     Starting event listener as pid 7702
  8 [15/May/2017 22:04:57 +0000] 7702 MainThread supervisor_listener INFO     Cannot open agent FIFO (agent probably dead), dropping event
  9 [15/May/2017 22:06:34 +0000] 7874 MainThread supervisor_listener INFO     Starting event listener as pid 7874
 10 [15/May/2017 22:06:35 +0000] 7874 MainThread supervisor_listener INFO     Cannot open agent FIFO (agent probably dead), dropping event

Following is /var/log/cloudera-scm-agent/cloudera-flood.log

 

  1 [12/May/2017 16:55:50 +0000] 45231 MainThread depot        INFO     Expiring torrents after 86400 seconds.
  2 [12/May/2017 16:55:50 +0000] 45231 MainThread server       INFO     Listen port: 7191
  3 [12/May/2017 16:55:50 +0000] 45231 Thread-2 server       INFO     listen_succeeded_alert successfully listening on [TCP] 0.0.0.0:7191
  4 [12/May/2017 16:55:50 +0000] 45231 Thread-2 server       INFO     listen_succeeded_alert successfully listening on [TCP/SSL] 0.0.0.0:4433
  5 [12/May/2017 16:55:50 +0000] 45231 Thread-2 server       INFO     listen_succeeded_alert successfully listening on [TCP] [::]:7191
  6 [12/May/2017 16:55:50 +0000] 45231 Thread-2 server       INFO     listen_succeeded_alert successfully listening on [TCP/SSL] [::]:4434
  7 [12/May/2017 16:55:50 +0000] 45231 Thread-2 server       INFO     listen_succeeded_alert successfully listening on [UDP] 0.0.0.0:7191
  8 [12/May/2017 16:55:50 +0000] 45231 Thread-2 server       INFO     dht_bootstrap_alert DHT bootstrap complete

 

Please help me out with this. Thanks in advance!

3 REPLIES 3

Re: Installation failed. Failed to receive heartbeat. Configured hostname, DNS and FQDN correctly.

New Contributor
I changed permission of log file and SSL and it worked. Thanks!

Re: Installation failed. Failed to receive heartbeat. Configured hostname, DNS and FQDN correctly.

New Contributor

Which log files did you change?  And to what permissions?  

 

What do you mean, you changed permission of SSL?  Could you explain please.

Highlighted

Re: Installation failed. Failed to receive heartbeat. Configured hostname, DNS and FQDN correctly.

New Contributor

I have the  sam problem, can you help me?