Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎01-09-2017

Error while setting up single node cluster using Cloudera Manager

While trying to setup a single node cluster, I am facing a common error for which I have seen a number of topics on the community, but I'm still not able to resolve the issue, here is the issue: 

 

When trying to install CDH 5.9.0 via Cloudera Manager, you may sometimes encounter "heartbeat" errors similar to the following:

 

    Installation failed. Failed to receive heartbeat from agent.

    Ensure that the host's hostname is configured properly.

    Ensure that port 7182 is accessible on the Cloudera Manager server (check firewall rules).

    Ensure that ports 9000 and 9001 are free on the host being added.

    Check agent logs in /var/log/cloudera-scm-agent/ on the host being added (some of the logs can be found in the installation details).

 

 

etc/hosts file:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.1.4 Cloudera01.******.cloudapp.net master.cdh-hadoop

 

 

here's what all I have done:

1. disabled SElinux

2. setenforce 

3. Opened port 7180 and 7182 to public

4. Created ssh-keygen and copied it into authorized_keys. Chmod 600 authorized_keys

 

 

 

Detailed error:


scm agent is already configured

Starting Cloudera Manager Agent...

BEGIN /sbin/service cloudera-scm-agent status
● cloudera-scm-agent.service - LSB: Cloudera SCM Agent
Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled)
Active: active (exited) since Mon 2017-01-09 19:42:12 UTC; 3min 14s ago
Docs: man:systemd-sysv-generator(8)
END (3)
BEGIN sudo /sbin/service cloudera-scm-agent start
Starting cloudera-scm-agent (via systemctl): [ OK ]
END (0)
agent logs:
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/'
>>[06/Jan/2017 22:20:10 +0000] 5192 MainThread agent INFO SCM Agent Version: 5.9.0
>>[06/Jan/2017 22:20:10 +0000] 5192 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[06/Jan/2017 22:20:10 +0000] 5192 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[06/Jan/2017 22:28:49 +0000] 8015 MainThread agent INFO SCM Agent Version: 5.9.0
>>[06/Jan/2017 22:28:49 +0000] 8015 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[06/Jan/2017 22:28:49 +0000] 8015 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 15:50:50 +0000] 51943 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 15:50:50 +0000] 51943 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 15:50:50 +0000] 51943 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 16:37:06 +0000] 56026 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 16:37:06 +0000] 56026 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 16:37:06 +0000] 56026 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 16:44:19 +0000] 1084 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 16:44:19 +0000] 1084 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 16:44:19 +0000] 1084 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 16:47:53 +0000] 3637 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 16:47:53 +0000] 3637 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 16:47:53 +0000] 3637 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 17:07:07 +0000] 6810 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 17:07:07 +0000] 6810 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 17:07:07 +0000] 6810 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 17:13:52 +0000] 1081 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 17:13:52 +0000] 1081 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 17:13:52 +0000] 1081 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 17:18:01 +0000] 3905 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 17:18:01 +0000] 3905 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 17:18:01 +0000] 3905 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 17:25:03 +0000] 6360 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 17:25:03 +0000] 6360 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 17:25:03 +0000] 6360 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 18:29:33 +0000] 11165 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 18:29:33 +0000] 11165 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 18:29:33 +0000] 11165 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 19:20:47 +0000] 15804 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 19:20:47 +0000] 15804 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 19:20:47 +0000] 15804 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 19:28:34 +0000] 18274 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 19:28:34 +0000] 18274 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 19:28:34 +0000] 18274 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 19:42:12 +0000] 19091 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 19:42:12 +0000] 19091 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 19:42:12 +0000] 19091 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 19:45:28 +0000] 21382 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 19:45:28 +0000] 21382 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 19:45:28 +0000] 21382 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
END (0)
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/'
>>[06/Jan/2017 22:20:10 +0000] 5192 MainThread agent INFO SCM Agent Version: 5.9.0
>>[06/Jan/2017 22:20:10 +0000] 5192 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[06/Jan/2017 22:20:10 +0000] 5192 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[06/Jan/2017 22:28:49 +0000] 8015 MainThread agent INFO SCM Agent Version: 5.9.0
>>[06/Jan/2017 22:28:49 +0000] 8015 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[06/Jan/2017 22:28:49 +0000] 8015 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 15:50:50 +0000] 51943 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 15:50:50 +0000] 51943 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 15:50:50 +0000] 51943 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 16:37:06 +0000] 56026 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 16:37:06 +0000] 56026 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 16:37:06 +0000] 56026 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 16:44:19 +0000] 1084 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 16:44:19 +0000] 1084 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 16:44:19 +0000] 1084 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 16:47:53 +0000] 3637 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 16:47:53 +0000] 3637 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 16:47:53 +0000] 3637 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 17:07:07 +0000] 6810 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 17:07:07 +0000] 6810 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 17:07:07 +0000] 6810 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 17:13:52 +0000] 1081 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 17:13:52 +0000] 1081 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 17:13:52 +0000] 1081 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 17:18:01 +0000] 3905 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 17:18:01 +0000] 3905 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 17:18:01 +0000] 3905 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 17:25:03 +0000] 6360 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 17:25:03 +0000] 6360 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 17:25:03 +0000] 6360 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 18:29:33 +0000] 11165 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 18:29:33 +0000] 11165 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 18:29:33 +0000] 11165 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 19:20:47 +0000] 15804 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 19:20:47 +0000] 15804 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 19:20:47 +0000] 15804 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 19:28:34 +0000] 18274 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 19:28:34 +0000] 18274 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 19:28:34 +0000] 18274 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 19:42:12 +0000] 19091 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 19:42:12 +0000] 19091 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 19:42:12 +0000] 19091 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[09/Jan/2017 19:45:28 +0000] 21382 MainThread agent INFO SCM Agent Version: 5.9.0
>>[09/Jan/2017 19:45:28 +0000] 21382 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[09/Jan/2017 19:45:28 +0000] 21382 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
>> return m2.ssl_connect(self.ssl)
>>SSLError: unknown protocol
>>[09/Jan/2017 19:45:15 +0000] 19103 MainThread agent ERROR Heartbeating to 10.0.1.4:7182 failed.
 
 
>>Traceback (most recent call last):
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/agent.py", line 1346, in _send_heartbeat
>> self.max_cert_depth)
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 132, in __init__
>> self.conn.connect()
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/httpslib.py", line 50, in connect
>> self.sock.connect((self.host, self.port))
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 185, in connect
>> ret = self.connect_ssl()
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
>> return m2.ssl_connect(self.ssl)
>>SSLError: unknown protocol
>>[09/Jan/2017 19:45:27 +0000] 19103 MainThread agent INFO Stopping agent...
>>[09/Jan/2017 19:45:27 +0000] 19103 MainThread agent INFO No extant cgroups; unmounting any cgroup roots
>>[09/Jan/2017 19:45:27 +0000] 19103 MainThread agent INFO No processes are being managed; Supervisor will shutdown.
>>[09/Jan/2017 19:45:27 +0000] 19103 MainThread agent INFO Shutting down supervisord, pid 19119
>>[09/Jan/2017 19:45:28 +0000] 19103 MainThread util INFO waiting for process to terminate...
>>[09/Jan/2017 19:45:28 +0000] 19103 MainThread util INFO Successfully killed process with pid 19119
>>[09/Jan/2017 19:45:28 +0000] 19103 MainThread _cplogging INFO [09/Jan/2017:19:45:28] ENGINE Bus STOPPING
>>[09/Jan/2017 19:45:28 +0000] 19103 MainThread _cplogging INFO [09/Jan/2017:19:45:28] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('Cloudera01.5ur24vnbopcepfnzrp30zx4shg.bx.internal.cloudapp.net', 9000)) shut down

 

 

>[09/Jan/2017 19:45:28 +0000] 21394 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation.
>>[09/Jan/2017 19:45:28 +0000] 21394 MainThread agent INFO Agent Logging Level: INFO
>>[09/Jan/2017 19:45:28 +0000] 21394 MainThread agent INFO No command line vars
>>[09/Jan/2017 19:45:28 +0000] 21394 MainThread agent INFO Missing database jar: /usr/share/java/mysql-connector-java.jar (normal, if you're not using this database type)
>>[09/Jan/2017 19:45:28 +0000] 21394 MainThread agent INFO Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type)
>>[09/Jan/2017 19:45:28 +0000] 21394 MainThread agent INFO Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
>>[09/Jan/2017 19:45:28 +0000] 21394 MainThread agent INFO Agent starting as pid 21394 user root(0) group root(0).
>>[09/Jan/2017 19:45:28 +0000] 21394 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/cgroups
>>[09/Jan/2017 19:45:28 +0000] 21394 MainThread cgroups INFO Found existing subsystem memory at /sys/fs/cgroup/memory
>>[09/Jan/2017 19:45:28 +0000] 21394 MainThread cgroups INFO Found existing subsystem cpuacct at /sys/fs/cgroup/cpu,cpuacct
>>[09/Jan/2017 19:45:28 +0000] 21394 MainThread cgroups INFO Found existing subsystem cpu at /sys/fs/cgroup/cpu,cpuacct
>>[09/Jan/2017 19:45:28 +0000] 21394 MainThread cgroups INFO Found existing subsystem blkio at /sys/fs/cgroup/blkio
END (0)
end of agent logs.
scm agent started

Installation script completed successfully.

all done
closing logging file descriptor
Cloudera Employee
Posts: 48
Registered: ‎08-16-2016

Re: Error while setting up single node cluster using Cloudera Manager

  • You might want to remove the firewall for internal communication. 
  • Might allow port 9000/9001 on the hosts.

 

This document has some more port information : http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_ports_cm.html#xd_583c10bfdbd326...