Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error: Installation failed. Failed to receive heartbeat from agent.

Error: Installation failed. Failed to receive heartbeat from agent.

New Contributor

Hi have the problem mentioned in the title.

I have disabled my firewall using

sudo ufw disable

but still i keep getting this error

 

This what the agent log file looks like

 

agent logs:
BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/'
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO SCM Agent Version: 5.4.1
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Not starting a new session.
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Adding env vars that start with CMF_AGENT_
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent        ERROR    Could not open log file; writing to stderr.
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 3142, in setup_rolling_logging
>> maxBytes=MAX_LOG_SIZE_BYTES, backupCount=LOG_BACKUP_COUNT)
>> File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__
>> BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
>> File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__
>> logging.FileHandler.__init__(self, filename, mode, encoding, delay)
>> File "/usr/lib/python2.7/logging/__init__.py", line 903, in __init__
>> StreamHandler.__init__(self, self._open())
>> File "/usr/lib/python2.7/logging/__init__.py", line 928, in _open
>> stream = open(self.baseFilename, self.mode)
>>IOError: [Errno 13] Permission denied: '/var/log/cloudera-scm-agent/cloudera-scm-agent.log'
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO SCM Agent Version: 5.4.1
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Agent Protocol Version: 4
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Using Host ID: 0a9aa26d-f528-4fe4-9756-3bd7445386a9
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Using directory: /run/cloudera-scm-agent
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Using supervisor binary path: /usr/lib/cmf/agent/src/cmf/../../build/env/bin/supervisord
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation.
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Agent Logging Level: INFO
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO No command line vars
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Found database jar: /usr/share/java/mysql-connector-java.jar
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type)
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Agent starting as pid 25533 user cloudera-scm(124) group cloudera-scm(134).
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Because agent not running as root, all processes will run with current user.
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[27/May/2015 11:47:28 +0000] 25533 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/cgroups
>>[27/May/2015 11:47:28 +0000] 25533 MainThread cgroups      WARNING  Could not read capabilities from /proc/cgroups. error: sudo: no tty present and no askpass program specified
>>
>>[27/May/2015 11:47:28 +0000] 25533 MainThread agent INFO No cgroups capabilities found
>>[27/May/2015 11:47:28 +0000] 25533 MainThread agent INFO Setting up supervisord event monitor.
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 3197, in <module>
>> main()
>> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 3180, in main
>> agent.configure()
>> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 477, in configure
>> self.event_mon = SupervisordEventMonitor(self.event_pipe_path)
>> File "/usr/lib/cmf/agent/src/cmf/supervisor_listener.py", line 112, in __init__
>> self.fifo = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
>>OSError: [Errno 13] Permission denied: '/run/cloudera-scm-agent/events'
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO SCM Agent Version: 5.4.1
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Not starting a new session.
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Adding env vars that start with CMF_AGENT_
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent        ERROR    Could not open log file; writing to stderr.
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 3142, in setup_rolling_logging
>> maxBytes=MAX_LOG_SIZE_BYTES, backupCount=LOG_BACKUP_COUNT)
>> File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__
>> BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
>> File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__
>> logging.FileHandler.__init__(self, filename, mode, encoding, delay)
>> File "/usr/lib/python2.7/logging/__init__.py", line 903, in __init__
>> StreamHandler.__init__(self, self._open())
>> File "/usr/lib/python2.7/logging/__init__.py", line 928, in _open
>> stream = open(self.baseFilename, self.mode)
>>IOError: [Errno 13] Permission denied: '/var/log/cloudera-scm-agent/cloudera-scm-agent.log'
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO SCM Agent Version: 5.4.1
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Agent Protocol Version: 4
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Using Host ID: 0a9aa26d-f528-4fe4-9756-3bd7445386a9
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Using directory: /run/cloudera-scm-agent
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Using supervisor binary path: /usr/lib/cmf/agent/src/cmf/../../build/env/bin/supervisord
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation.
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Agent Logging Level: INFO
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO No command line vars
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Found database jar: /usr/share/java/mysql-connector-java.jar
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type)
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Agent starting as pid 25533 user cloudera-scm(124) group cloudera-scm(134).
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Because agent not running as root, all processes will run with current user.
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[27/May/2015 11:47:26 +0000] 25533 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[27/May/2015 11:47:28 +0000] 25533 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/cgroups
>>[27/May/2015 11:47:28 +0000] 25533 MainThread cgroups      WARNING  Could not read capabilities from /proc/cgroups. error: sudo: no tty present and no askpass program specified
>>
>>[27/May/2015 11:47:28 +0000] 25533 MainThread agent INFO No cgroups capabilities found
>>[27/May/2015 11:47:28 +0000] 25533 MainThread agent INFO Setting up supervisord event monitor.
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 3197, in <module>
>> main()
>> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 3180, in main
>> agent.configure()
>> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 477, in configure
>> self.event_mon = SupervisordEventMonitor(self.event_pipe_path)
>> File "/usr/lib/cmf/agent/src/cmf/supervisor_listener.py", line 112, in __init__
>> self.fifo = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
>>OSError: [Errno 13] Permission denied: '/run/cloudera-scm-agent/events'
END (0)
BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/'
>>[27/May/2015 10:15:54 +0000] 2141 MainThread agent INFO Loading last saved hb response to complete initialization: /var/lib/cloudera-scm-agent/response.avro
>>[27/May/2015 10:15:54 +0000] 2141 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels
>>[27/May/2015 10:15:54 +0000] 2141 MainThread parcel INFO Agent does create users/groups and apply file permissions
>>[27/May/2015 10:15:54 +0000] 2141 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
>>[27/May/2015 10:15:54 +0000] 2141 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
>>[27/May/2015 10:15:55 +0000] 2141 MainThread agent INFO Active parcel list updated; recalculating component info.
>>[27/May/2015 10:15:59 +0000] 2141 Monitor-HostMonitor throttling_logger INFO Using java location: '/usr/lib/jvm/java-7-oracle-cloudera/bin/java'.
>>[27/May/2015 10:15:59 +0000] 2141 Monitor-HostMonitor throttling_logger ERROR    Failed to collect NTP metrics
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/src/cmf/monitor/host/ntp_monitor.py", line 37, in collect
>> result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout)
>> File "/usr/lib/cmf/agent/src/cmf/monitor/host/ntp_monitor.py", line 30, in _subprocess_with_timeout
>> return subprocess_with_timeout(args, timeout)
>> File "/usr/lib/cmf/agent/src/cmf/subprocess_timeout.py", line 49, in subprocess_with_timeout
>> p = subprocess.Popen(**kwargs)
>> File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
>> errread, errwrite)
>> File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
>> raise child_exception
>>OSError: [Errno 2] No such file or directory
>>[27/May/2015 10:46:00 +0000] 2141 Monitor-HostMonitor throttling_logger ERROR    (359 skipped) Failed to collect NTP metrics
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/src/cmf/monitor/host/ntp_monitor.py", line 37, in collect
>> result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout)
>> File "/usr/lib/cmf/agent/src/cmf/monitor/host/ntp_monitor.py", line 30, in _subprocess_with_timeout
>> return subprocess_with_timeout(args, timeout)
>> File "/usr/lib/cmf/agent/src/cmf/subprocess_timeout.py", line 49, in subprocess_with_timeout
>> p = subprocess.Popen(**kwargs)
>> File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
>> errread, errwrite)
>> File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
>> raise child_exception
>>OSError: [Errno 2] No such file or directory
>>[27/May/2015 11:04:32 +0000] 2141 MainThread agent INFO Stopping agent...
>>[27/May/2015 11:04:32 +0000] 2141 MainThread agent INFO No extant cgroups; unmounting any cgroup roots
>>[27/May/2015 11:04:32 +0000] 2141 MainThread agent INFO No processes are being managed; Supervisor will shutdown.
>>[27/May/2015 11:04:32 +0000] 2141 MainThread agent INFO Shutting down supervisord, pid 2183
>>[27/May/2015 11:04:33 +0000] 2141 MainThread agent INFO waiting for process to terminate...
>>[27/May/2015 11:04:33 +0000] 2141 MainThread agent INFO Successfully killed process with pid 2183
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus STOPPING
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('axel', 9000)) shut down
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Stopped thread '_TimeoutMonitor'.
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus STOPPED
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus STOPPING
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('axel', 9000)) already shut down
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE No thread running for None.
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus STOPPED
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus EXITING
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus EXITED
>>[27/May/2015 11:04:33 +0000] 2141 MainThread agent INFO Agent exiting; caught signal 15
>>[27/May/2015 10:15:54 +0000] 2141 MainThread agent INFO Loading last saved hb response to complete initialization: /var/lib/cloudera-scm-agent/response.avro
>>[27/May/2015 10:15:54 +0000] 2141 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels
>>[27/May/2015 10:15:54 +0000] 2141 MainThread parcel INFO Agent does create users/groups and apply file permissions
>>[27/May/2015 10:15:54 +0000] 2141 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
>>[27/May/2015 10:15:54 +0000] 2141 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
>>[27/May/2015 10:15:55 +0000] 2141 MainThread agent INFO Active parcel list updated; recalculating component info.
>>[27/May/2015 10:15:59 +0000] 2141 Monitor-HostMonitor throttling_logger INFO Using java location: '/usr/lib/jvm/java-7-oracle-cloudera/bin/java'.
>>[27/May/2015 10:15:59 +0000] 2141 Monitor-HostMonitor throttling_logger ERROR    Failed to collect NTP metrics
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/src/cmf/monitor/host/ntp_monitor.py", line 37, in collect
>> result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout)
>> File "/usr/lib/cmf/agent/src/cmf/monitor/host/ntp_monitor.py", line 30, in _subprocess_with_timeout
>> return subprocess_with_timeout(args, timeout)
>> File "/usr/lib/cmf/agent/src/cmf/subprocess_timeout.py", line 49, in subprocess_with_timeout
>> p = subprocess.Popen(**kwargs)
>> File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
>> errread, errwrite)
>> File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
>> raise child_exception
>>OSError: [Errno 2] No such file or directory
>>[27/May/2015 10:46:00 +0000] 2141 Monitor-HostMonitor throttling_logger ERROR    (359 skipped) Failed to collect NTP metrics
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/src/cmf/monitor/host/ntp_monitor.py", line 37, in collect
>> result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout)
>> File "/usr/lib/cmf/agent/src/cmf/monitor/host/ntp_monitor.py", line 30, in _subprocess_with_timeout
>> return subprocess_with_timeout(args, timeout)
>> File "/usr/lib/cmf/agent/src/cmf/subprocess_timeout.py", line 49, in subprocess_with_timeout
>> p = subprocess.Popen(**kwargs)
>> File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
>> errread, errwrite)
>> File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
>> raise child_exception
>>OSError: [Errno 2] No such file or directory
>>[27/May/2015 11:04:32 +0000] 2141 MainThread agent INFO Stopping agent...
>>[27/May/2015 11:04:32 +0000] 2141 MainThread agent INFO No extant cgroups; unmounting any cgroup roots
>>[27/May/2015 11:04:32 +0000] 2141 MainThread agent INFO No processes are being managed; Supervisor will shutdown.
>>[27/May/2015 11:04:32 +0000] 2141 MainThread agent INFO Shutting down supervisord, pid 2183
>>[27/May/2015 11:04:33 +0000] 2141 MainThread agent INFO waiting for process to terminate...
>>[27/May/2015 11:04:33 +0000] 2141 MainThread agent INFO Successfully killed process with pid 2183
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus STOPPING
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('axel', 9000)) shut down
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Stopped thread '_TimeoutMonitor'.
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus STOPPED
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus STOPPING
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('axel', 9000)) already shut down
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE No thread running for None.
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus STOPPED
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus EXITING
>>[27/May/2015 11:04:33 +0000] 2141 MainThread _cplogging INFO [27/May/2015:11:04:33] ENGINE Bus EXITED
>>[27/May/2015 11:04:33 +0000] 2141 MainThread agent INFO Agent exiting; caught signal 15
END (0)
end of agent logs.
scm agent started 

Do you have any idea what goes wrong?

4 REPLIES 4

Re: Error: Installation failed. Failed to receive heartbeat from agent.

- Is this a CM install with the single-user option?
- Is the Agent running as root or another user?
- Please provide the permissions of the /var/log/cloudera-scm-agent directory
- Does /usr/lib/jvm/java-7-oracle-cloudera/bin/java exist?
Regards,
Gautam Gopalakrishnan

Re: Error: Installation failed. Failed to receive heartbeat from agent.

New Contributor

Hi, thanks for the response.

 

- Yes it a Cm install with single user option enabled, but i also tried to install without it.Same problem

 

- The agent indeed is running as root

 

- The permissions are

 

-rw-r--r-- 1 root root 56399 May 27 11:04 cloudera-scm-agent.log
-rw-r--r-- 1 root root 10991 May 28 10:15 cloudera-scm-agent.out
-rw-r--r-- 1 root root   612 May 27 10:15 cmf_listener.log
-rw-r--r-- 1 root root  2233 May 27 11:04 supervisord.log
-rw-r--r-- 1 root root   440 May 27 10:15 supervisord.out

- Yes it does exist

 

Highlighted

Re: Error: Installation failed. Failed to receive heartbeat from agent.

New Contributor

Yami,

 

were you able to resolve your issue? I am also facing the same problem. not sure how to fix it. let meknow if you already fix this problem.

Re: Error: Installation failed. Failed to receive heartbeat from agent.

New Contributor

Hi,

 Their could be various reason for this error. I have also received the error.

 

Reslution-1 :  I have resolved this error by editing the hosts file.

We need to have below entry for all nodes in cluster on all hosts file in cluster.

 

<IP> <FQDN> <SHORTNAME>

 

Resolution 2 : I have mentioned the each hostname in new line during the installation. However, if you go back to the window where you mentioned ll hos

and try installing again your installation will be sucessfull. Else, try togiv the hostname comma seperated at the first-place.

 

I hope it helps.

 

Regards,

Neeraj