Reply
New Contributor
Posts: 10
Registered: ‎08-09-2017

Re: Cloudera Manager. Agent cannot connect with supervisor. Failed to receive heartbeat from agent

[root@bigdata-master cloudera-scm-agent]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Tue Aug 8 03:48:20 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=6a383e2f-6ebf-4654-b3ef-cd7917564a78 / xfs defaults 0 0
UUID=b347299f-6786-4f12-a3eb-84d580423ebc /boot xfs defaults 0 0
UUID=6523647e-3167-4caf-a3cc-135dc081b1b8 swap swap defaults 0 0

 

this is my fstab file content,i am not sure where the problem is.

Champion
Posts: 735
Registered: ‎05-16-2016

Re: Cloudera Manager. Agent cannot connect with supervisor. Failed to receive heartbeat from agent

could you share your /etc/hosts ? 

status of your firewall ? 

what os are you runining  ? 

 

New Contributor
Posts: 10
Registered: ‎08-09-2017

Re: Cloudera Manager. Agent cannot connect with supervisor. Failed to receive heartbeat from agent

thus,i have post a new topic at https://community.cloudera.com/t5/Cloudera-Manager-Installation/CDH-5-12-0-clouder-manager-agent-can... infomation is post on this url.thks for help.

New Contributor
Posts: 2
Registered: ‎04-11-2018

Re: Cloudera Manager. Agent cannot connect with supervisor. Failed to receive heartbeat from agent

To resolve this issue you need to check first all port opened on your server service listing to the port no, use command - sudo netstat -lpten check if any thing is running on 9000 or 90001, mostly java services required for set up is running on port 9000, and cloudera-scm-agent listner also runs on port 9000. to over come this issue you can re-configure theports as well in /etc/cloudera-scm-agent/config.ini by changing as below:-

 

# Hostname of the CM server.
server_host=10.34.156.118

# Port that the CM server is listening on.
server_port=7182

## It should not normally be necessary to modify these.
# Port that the CM agent should listen on.
listening_port=9001

 

It should not normally be necessary to modify these. # Port that the CM agent should listen on 9001

and then restart the cloudera-scm-agent service by command:- $ service cloudera-scm-agent restart TO verify this port is not activated for other sshd service check Ports opened in /etc/ssh/sshd_config.

 

I hope this resolution will work for others too.

Cheers,

Ankit Gupta

New Contributor
Posts: 5
Registered: ‎01-25-2017

Re: Cloudera Manager. Agent cannot connect with supervisor. Failed to receive heartbeat from agent

Hi,

 

I am facing the same issue where, cloudera agent is exiting because it is unable to find the supervisord.conf file. The conf file is not getting created. 

 

Error see in cloudera-scm-agent.log file

 

[24/May/2018 14:19:31 +0000] 10889 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
/supervisor/include
[24/May/2018 14:19:31 +0000] 10889 MainThread agent ERROR Failed to connect to previous supervisor.
 Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2110, in find_or_start_supervisor
self.get_supervisor_process_info()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2254, in get_supervisor_process_info
self.identifier = self.supervisor_client.supervisor.getIdentification()
File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request
self.connection.request('POST', handler, request_body, self.headers)
File "/usr/lib64/python2.7/httplib.py", line 1017, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.7/httplib.py", line 1051, in _send_request
self.endheaders(body)
File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders
self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output
self.send(msg)
File "/usr/lib64/python2.7/httplib.py", line 826, in send
self.connect()
File "/usr/lib64/python2.7/httplib.py", line 807, in connect
self.timeout, self.source_address)
File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
raise err

error: [Errno 111] Connection refused
[24/May/2018 14:19:31 +0000] 10889 Dummy-1 daemonize WARNING Stopping daemon.
[24/May/2018 14:19:31 +0000] 10889 Dummy-1 agent INFO Stopping agent...
[24/May/2018 14:19:31 +0000] 10889 Dummy-1 agent INFO No extant cgroups; unmounting any cgroup roots

 

The cloudera manager and agent version is 5.12.0

 

We did have a server failure and after the reboot of the server, the agent does not start.

 

Ran the below the command:

/usr/lib64/cmf/agent/build/env/bin/python /usr/lib64/cmf/agent/build/env/bin/cmf-agent --package_dir /usr/lib64/cmf/service --agent_dir /var/run/cloudera-scm-agent --lib_dir /var/lib/cloudera-scm-agent --logfile /var/log/cloudera-scm-agent/cloudera-scm-agent.log --comm_name cmf-agent --pidfile /var/run/cloudera-scm-agent/cloudera-scm-agent.pid

 

Output:


[24/May/2018 14:27:45 +0000] 11274 MainThread agent INFO SCM Agent Version: 5.12.0
[24/May/2018 14:27:45 +0000] 11274 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
[24/May/2018 14:27:45 +0000] 11274 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
[24/May/2018 14:27:45 +0000] 11274 MainThread agent INFO Not starting a new session.
[24/May/2018 14:27:45 +0000] 11274 MainThread agent INFO Adding env vars that start with CMF_AGENT_
[24/May/2018 14:27:45 +0000] 11274 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/bin/cmf-agent", line 12, in <module>
load_entry_point('cmf==5.12.0', 'console_scripts', 'cmf-agent')()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 3095, in main
main_impl()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 3078, in main_impl
agent.start()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 804, in start
self.find_or_start_supervisor()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2151, in find_or_start_supervisor
if not mount_tmpfs(process_dir, self.args.clear_agent_dir, self.sudo):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/tmpfs.py", line 62, in mount_tmpfs
if os.path.samefile(p.mountpoint, path) and p.fstype == "tmpfs" and "noexec" not in p.opts:
File "/usr/lib64/cmf/agent/build/env/lib64/python2.7/posixpath.py", line 162, in samefile
s1 = os.stat(f1)
OSError: [Errno 2] No such file or directory: 'net:[4026532100]'

 

Please may you help on this? Is the issue fixed in v5.12.2?

Announcements