Reply
Highlighted
Explorer
Posts: 9
Registered: ‎08-07-2017
Accepted Solution

Missing supervisord.conf in /run/cloudera-scm-agent/supervisor/ directory

Hi, I am trying to install cloudera-manager 5.12.0, but during installation of agents in final stage I recieve error like this << Failed to receive heartbeat from agent. >>
In log file /var/log/cloudera-scm-agent/cloudera-scm-agent.log I found an error which said  << Failed to connect to previous supervisor. >>

I started to debug cloudera-scm-agent code and found that it fail when it is trying to find supervisord.conf  in  /run/cloudera-scm-agent/supervisor/ directory. 

In file called agent.py there is function called configure_supervisor_clients(self) and it fails on line 2 when it trying to create path for supervisord.conf because there is no such file in that path.

 

1 def configure_supervisor_clients(self):
    """
    Configure the supervisor client.
    We could do the XMLRPC configuration manually, but
    why bother if there's helper code to do it for us!
    """
    1. supervisor_options = supervisor.options.ClientOptions()

    2. supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")])
    3. self.supervisor_client = supervisor_options.getServerProxy()
    4. self.supervisor_ctl = supervisor.supervisorctl.Controller(supervisor_options)

I know that agent are responsible to create that conf file but it seems that it is not creating that file at all.


Can anyone help me with this problem ?
I do all my installations with cloudera-manager-installer.bin script.

 

New Contributor
Posts: 11
Registered: ‎08-07-2017

Re: Missing supervisord.conf in /run/cloudera-scm-agent/supervisor/ directory

I have the same question like this.
That's is log info blow :
[08/Aug/2017 08:28:55 +0000] 17951 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[08/Aug/2017 08:28:55 +0000] 17951 MainThread agent INFO Created /run/cloudera-scm-agent/process
[08/Aug/2017 08:28:55 +0000] 17951 MainThread agent INFO Chmod'ing /run/cloudera-scm-agent/process to 0751
[08/Aug/2017 08:28:55 +0000] 17951 MainThread agent INFO Created /run/cloudera-scm-agent/supervisor
[08/Aug/2017 08:28:55 +0000] 17951 MainThread agent INFO Chmod'ing /run/cloudera-scm-agent/supervisor to 0751
[08/Aug/2017 08:28:55 +0000] 17951 MainThread agent INFO Created /run/cloudera-scm-agent/flood
[08/Aug/2017 08:28:55 +0000] 17951 MainThread agent INFO Chowning /run/cloudera-scm-agent/flood to cloudera-scm (988) cloudera-scm (983)
[08/Aug/2017 08:28:55 +0000] 17951 MainThread agent INFO Chmod'ing /run/cloudera-scm-agent/flood to 0751
[08/Aug/2017 08:28:55 +0000] 17951 MainThread agent INFO Created /run/cloudera-scm-agent/supervisor/include
[08/Aug/2017 08:28:55 +0000] 17951 MainThread agent INFO Chmod'ing /run/cloudera-scm-agent/supervisor/include to 0751
[08/Aug/2017 08:28:55 +0000] 17951 MainThread agent ERROR Failed to connect to previous supervisor.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2109, in find_or_start_supervisor
self.configure_supervisor_clients()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2290, in configure_supervisor_clients
supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")])
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 1599, in realize
Options.realize(self, *arg, **kw)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 333, in realize
self.process_config()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 341, in process_config
self.process_config_file(do_usage)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 376, in process_config_file
self.usage(str(msg))
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 164, in usage
self.exit(2)
New Contributor
Posts: 11
Registered: ‎08-07-2017

Re: Missing supervisord.conf in /run/cloudera-scm-agent/supervisor/ directory

I have already checked the /etc/hosts file,/etc/sysconfig/network file and hostname, it's all ok.
it's likely the supervisor can't run when the agent can't generate the supervisor conf file.

New Contributor
Posts: 5
Registered: ‎08-09-2017

Re: Missing supervisord.conf in /run/cloudera-scm-agent/supervisor/ directory

[ Edited ]

Hi vigensahakyan, 

check if there are /run/cloudera-scm-agent/supervisor/supervisord.conf and .pid files and include/flood.conf on your failing host. If not, copy them from another host. Then run supervisord daemon and retry the installation.

Explorer
Posts: 9
Registered: ‎08-07-2017

Re: Missing supervisord.conf in /run/cloudera-scm-agent/supervisor/ directory

It's not working in that way as it trying to connect to supervisor which doesn't actually exist and give en error like this

[09/Aug/2017 14:40:01 +0000] 27188 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/process
[09/Aug/2017 14:40:01 +0000] 27188 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor
[09/Aug/2017 14:40:01 +0000] 27188 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/flood
[09/Aug/2017 14:40:01 +0000] 27188 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include
[09/Aug/2017 14:40:01 +0000] 27188 MainThread agent ERROR Failed to connect to previous supervisor.
Traceback (most recent call last):
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2110, in find_or_start_supervisor
self.get_supervisor_process_info()
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2254, in get_supervisor_process_info
self.identifier = self.supervisor_client.supervisor.getIdentification()
File "/usr/lib/python2.7/xmlrpclib.py", line 1243, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1602, in __request
verbose=self.__verbose
File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request
self.connection.request('POST', handler, request_body, self.headers)
File "/usr/lib/python2.7/httplib.py", line 1057, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1097, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 897, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 859, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 836, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 575, in create_connection
raise err
error: [Errno 111] Connection refused
[09/Aug/2017 14:40:01 +0000] 27188 Dummy-1 daemonize WARNING Stopping daemon.
[09/Aug/2017 14:40:01 +0000] 27188 Dummy-1 agent INFO Stopping agent...
[09/Aug/2017 14:40:01 +0000] 27188 Dummy-1 agent INFO No extant cgroups; unmounting any cgroup roots
New Contributor
Posts: 5
Registered: ‎08-09-2017

Re: Missing supervisord.conf in /run/cloudera-scm-agent/supervisor/ directory

Is anything listening on port 19001?
Use this to check:
netstat -anp | grep 19001
Explorer
Posts: 9
Registered: ‎08-07-2017

Re: Missing supervisord.conf in /run/cloudera-scm-agent/supervisor/ directory

No, there is nothing on port 19001 and 9000.
Actually it is fail to run supervisord, hence that I have nothing on port 19001.
New Contributor
Posts: 11
Registered: ‎08-07-2017

Re: Missing supervisord.conf in /run/cloudera-scm-agent/supervisor/ directory

I've coped supervisord.conf supervisord.pid and flood.conf to right dir, but it still doesn't work 

error info is different 

 

[09/Aug/2017 19:44:56 +0000] 8239 MainThread agent ERROR Failed to connect to previous supervisor.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2110, in find_or_start_supervisor
self.get_supervisor_process_info()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2254, in get_supervisor_process_info
self.identifier = self.supervisor_client.supervisor.getIdentification()
File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request
self.connection.request('POST', handler, request_body, self.headers)
File "/usr/lib64/python2.7/httplib.py", line 1017, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.7/httplib.py", line 1051, in _send_request
self.endheaders(body)
File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders
self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output
self.send(msg)
File "/usr/lib64/python2.7/httplib.py", line 826, in send
self.connect()
File "/usr/lib64/python2.7/httplib.py", line 807, in connect
self.timeout, self.source_address)
File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
raise err

New Contributor
Posts: 5
Registered: ‎08-09-2017

Re: Missing supervisord.conf in /run/cloudera-scm-agent/supervisor/ directory

Run supervisord.
New Contributor
Posts: 11
Registered: ‎08-07-2017

Re: Missing supervisord.conf in /run/cloudera-scm-agent/supervisor/ directory

how to run it?
Announcements