Support Questions
Find answers, ask questions, and share your expertise

Failed to receive heartbeat from agent

Failed to receive heartbeat from agent

Explorer

I'm new to the hadoop and cloudera, trying to learn. I'm trying to install cloudera manager on ubuntu 14.04 LTS on single node and I'm getting “Failed to receive heartbeat from agent” error. I tried several options from cloudera community messages, nothing worked for me. Below are the error details and my pc details. Could you please help me?

 

>> ret = self.connect_ssl()

>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
>> return m2.ssl_connect(self.ssl)
>>SSLError: unknown protocol
>>
[30/Sep/2016 18:59:17 +0000] 4297 MainThread agent ERROR Heartbeating to 192.168.0.5:7182 failed.
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/agent.py", line 1244, in _send_heartbeat
>> self.max_cert_depth)
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/https.py", line 132, in __init__
>> self.conn.connect()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/httpslib.py", line 50, in connect
>> self.sock.connect((self.host, self.port))
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 185, in connect
>> ret = self.connect_ssl()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
>> return m2.ssl_connect(self.ssl)
>>
SSLError: unknown protocol
>>[30/Sep/2016 18:59:21 +0000] 4297 MainThread agent INFO Stopping agent...
>>[30/Sep/2016 18:59:21 +0000] 4297 MainThread agent INFO No processes are being managed; Supervisor will shutdown.
>>[30/Sep/2016 18:59:21 +0000] 4297 MainThread agent INFO Shutting down supervisord, pid 4336
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread agent INFO waiting for process to terminate...
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread agent INFO Successfully killed process with pid 4336
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread _cplogging INFO [30/Sep/2016:18:59:22] ENGINE Bus STOPPING
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread _cplogging INFO [30/Sep/2016:18:59:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('venu-HP-Pavilion-Sleekbook-14-PC.Home', 9000)) shut down
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread _cplogging INFO [30/Sep/2016:18:59:22] ENGINE Stopped thread '_TimeoutMonitor'.
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread _cplogging INFO [30/Sep/2016:18:59:22] ENGINE Bus STOPPED
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread _cplogging INFO [30/Sep/2016:18:59:22] ENGINE Bus STOPPING
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread _cplogging INFO [30/Sep/2016:18:59:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('venu-HP-Pavilion-Sleekbook-14-PC.Home', 9000)) already shut down
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread _cplogging INFO [30/Sep/2016:18:59:22] ENGINE No thread running for None.
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread _cplogging INFO [30/Sep/2016:18:59:22] ENGINE Bus STOPPED
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread _cplogging INFO [30/Sep/2016:18:59:22] ENGINE Bus EXITING
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread _cplogging INFO [30/Sep/2016:18:59:22] ENGINE Bus EXITED
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread agent INFO Agent exiting; caught signal 15
>>[30/Sep/2016 18:59:23 +0000] 5487 MainThread agent INFO SCM Agent Version: 5.8.2
>>[30/Sep/2016 18:59:23 +0000] 5487 MainThread agent INFO Agent Protocol Version: 4
>>[30/Sep/2016 18:59:23 +0000] 5487 MainThread agent INFO Using Host ID: 7a0bddde-5342-4a7e-be55-ef3a2b7880a0
>>[30/Sep/2016 18:59:23 +0000] 5487 MainThread agent INFO Using directory: /run/cloude>> self.conn.connect()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/httpslib.py", line 50, in connect
>> self.sock.connect((self.host, self.port))
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 185, in connect
>> ret = self.connect_ssl()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
>> return m2.ssl_connect(self.ssl)
>>
SSLError: unknown protocol
>>[30/Sep/2016 18:59:17 +0000] 4297 MainThread agent ERROR Heartbeating to 192.168.0.5:7182 failed.
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/agent.py", line 1244, in _send_heartbeat
>> self.max_cert_depth)
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/https.py", line 132, in __init__
>> self.conn.connect()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/httpslib.py", line 50, in connect
>> self.sock.connect((self.host, self.port))
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 185, in connect
>> ret = self.connect_ssl()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
>> return m2.ssl_connect(self.ssl)
>
>SSLError: unknown protocol
>>[30/Sep/2016 18:59:21 +0000] 4297 MainThread agent INFO Stopping agent...
>>[30/Sep/2016 18:59:21 +0000] 4297 MainThread agent INFO No processes are being managed; Supervisor will shutdown.
>>[30/Sep/2016 18:59:21 +0000] 4297 MainThread agent INFO Shutting down supervisord, pid 4336
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread agent INFO waiting for process to terminate...
>>[30/Sep/2016 18:59:22 +0000] 4297 MainThread agent INFO Successfully killed process with pid 4336

 

my /etc/hosts

 

127.0.0.1 localhost.hadoopdomain localhost

192.168.0.5 venu-HP-Pavilion-Sleekbook-14-PC.Home venu-HP-Pavilion-Sleekbook-14$

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

 

 

My /etc/hostname file

 

venu-HP-Pavilion-Sleekbook-14-PC

 

my hostname and hostname -f

 

venu@venu-HP-Pavilion-Sleekbook-14-PC:~$ hostname

venu-HP-Pavilion-Sleekbook-14-PC

venu@venu-HP-Pavilion-Sleekbook-14-PC:~$ hostname -f

venu-HP-Pavilion-Sleekbook-14-PC.Home

venu@venu-HP-Pavilion-Sleekbook-14-PC:~$

 

firewall is also off

 

venu@venu-HP-Pavilion-Sleekbook-14-PC:~$ sysv-rc-conf –list

 

acpid

anacron

apache2 0:off 1:off 6:off

apparmor S:on

apport

avahi-daemon

binfmt-suppo

bluetooth

brltty S:on

cloudera-scm 2:off 3:off 4:off 5:off

cloudera-scm 0:off 1:off 2:on 3:on 4:on 5:on 6:off

cloudera-scm 0:off 1:off 2:on 3:on 4:on 5:on 6:off

cloudera-scm 0:off 1:off 2:on 3:on 4:on 5:on 6:off

console-setu

 

while installing one of the screens (below screen) I tried with both ip address 192.168.0.5 and 127.0.0.1

Sreen name Specify hosts for your CDH cluster installation.

 

My ifconfig -a returns following

 

venu@venu-HP-Pavilion-Sleekbook-14-PC:~$ ifconfig -a

 

eth0 Link encap:Ethernet HWaddr 84:34:97:78:ee:7a

UP BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

 

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:57982 errors:0 dropped:0 overruns:0 frame:0

TX packets:57982 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:41652208 (41.6 MB) TX bytes:41652208 (41.6 MB)

 

wlan0 Link encap:Ethernet HWaddr 20:68:9d:ce:55:a4

inet addr:192.168.0.5 Bcast:192.168.0.255 Mask:255.255.255.0

inet6 addr: fd00::2268:9dff:fece:55a4/64 Scope:Global

inet6 addr: fd00::e9da:918:329f:633c/64 Scope:Global

inet6 addr: fe80::2268:9dff:fece:55a4/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:92129 errors:0 dropped:0 overruns:0 frame:0

TX packets:72717 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:93448735 (93.4 MB) TX bytes:18114107 (18.1 MB)

 

 

I believe port 9000 is open based on below nmap command

 

venu@venu-HP-Pavilion-Sleekbook-14-PC:~$ nmap 192.168.0.5

 

Starting Nmap 6.40 ( http://nmap.org ) at 2016-09-30 20:22 MDT

Nmap scan report for venu-HP-Pavilion-Sleekbook-14-PC.Home (192.168.0.5)

Host is up (0.00021s latency).

Not shown: 995 closed ports

PORT STATE SERVICE

22/tcp open ssh

111/tcp open rpcbind

139/tcp open netbios-ssn

445/tcp open microsoft-ds

9000/tcp open cslistener

 

Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds

 

My telnet 192.168.0.5 7182 also successful

 

After each change I'm restarting cloudera-scm-agent, sometimes I'm rebooting my system.

4 REPLIES 4

Re: Failed to receive heartbeat from agent

Cloudera Employee

Since this is a single node, can you perform the following edits?:

 

1. Edit the /etc/hosts file:

 

Change the entry from:

 

127.0.0.1 localhost.hadoopdomain localhost

 

to

 

127.0.0.1 localhost localhost.hadoopdomain

2. Edit the /etc/cloudera-scm-agent/config.ini file by either confirming or changing the entry (third line from the top) to the following:

 

server_host=localhost

 

Re: Failed to receive heartbeat from agent

Explorer

 Hi Rufusayeni

 

Thank you so much for your time.

 

I tried with your suggestion but I got the same error.

 

I didn't get any other option. So, I uninstalled cloudera manager and insatalled it again. I din't get this error. At least I move forward upto cluster setup. Now, I'm getting different errors with cluster setup. I'm working on this now. these are different errors. May be I'll post it in a different thread. Thank you again.

 

Regards

Vga

Re: Failed to receive heartbeat from agent

Cloudera Employee

You are welcome.

Re: Failed to receive heartbeat from agent

Explorer

Hi 

 

I am facing the same issue. One node in cluster is not sending heartbeat. 

 

MainThread agent ERROR Heartbeating to XXXX:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1399, in _send_heartbeat
response = self.requestor.request('heartbeat', heartbeat_data)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
return self.issue_request(call_request, message_name, request_datum)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
call_response = self.transceiver.transceive(call_request)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 482, in transceive
self.write_framed_message(request)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 501, in write_framed_message
self.conn.request(req_method, self.req_resource, req_body, req_headers)
File "/usr/lib64/python2.7/httplib.py", line 1041, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
self.endheaders(body)
File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
self.send(msg)
File "/usr/lib64/python2.7/httplib.py", line 857, in send
self.sock.sendall(data)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 351, in write
return self._write_bio(data)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 330, in _write_bio
return m2.ssl_write(self.ssl, data, self._timeout)
SSLError: (32, 'Broken pipe')
[29/Jul/2019 04:05:42 +0000] 6869 MainThread agent ERROR Heartbeating to XXXX:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1390, in _send_heartbeat

 

Could you help us to resolve the issue.