Support Questions
Find answers, ask questions, and share your expertise

Installation failed. Failed to receive heartbeat on cluster installation on singleNode from agents.

New Contributor

I am trying to install cloudera CDH 5.14.0. and facing below error during Cluster Installation.

Installation failed. Failed to receive heartbeat from agent.

  • Ensure that the host's hostname is configured properly.
  • Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
  • Ensure that ports 9000 and 9001 are not in use on the host being added.
  • Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).
  • If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.

 

Delatil Log:

/tmp/scm_prepare_node.5elWQ7Qo 
using SSH_CLIENT to get the SCM hostname: 127.0.0.1 55972 22 
opening logging file descriptor 
Starting installation script...
Acquiring installation lock...
BEGIN flock 4 
END (0) 
Detecting root privileges...
effective UID is 1000 
BEGIN which pbrun 
END (1) 
BEGIN sudo -S id 
uid=0(root) gid=0(root) groups=0(root) 
END (0) 
Using 'sudo ' to acquire root privileges 
Detecting distribution...
BEGIN grep 'Ubuntu' /etc/lsb-release 
DISTRIB_ID=Ubuntu 
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS" 
END (0) 
BEGIN grep DISTRIB_CODENAME /etc/lsb-release 
DISTRIB_CODENAME=xenial 
END (0) 
BEGIN echo DISTRIB_CODENAME=xenial | cut -d = -f 2 
END (0) 
xenial 
Detecting Cloudera Manager Server...
BEGIN host -t PTR 127.0.0.1 
Host 1.0.0.127.in-addr.arpa. not found: 3(NXDOMAIN) 
END (1) 
BEGIN which python 
/usr/bin/python 
END (0) 
BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' 127.0.0.1 7182 
END (0) 
BEGIN which wget 
/usr/bin/wget 
END (0) 
BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo 
END (4) 
Installing package repositories...
validating format of repository file /tmp/scm_prepare_node.5elWQ7Qo/repos/ubuntu_xenial/cloudera-manager.list 
installing repository file /tmp/scm_prepare_node.5elWQ7Qo/repos/ubuntu_xenial/cloudera-manager.list 
repository file /tmp/scm_prepare_node.5elWQ7Qo/repos/ubuntu_xenial/cloudera-manager.list installed 
BEGIN sudo apt-key add /tmp/scm_prepare_node.5elWQ7Qo/archive.key 
OK 
END (0) 
installing priority file /tmp/scm_prepare_node.5elWQ7Qo/priorities/ubuntu_xenial/cloudera.pref 
priority file /tmp/scm_prepare_node.5elWQ7Qo/priorities/ubuntu_xenial/cloudera.pref installed 
Refreshing package metadata...
BEGIN sudo apt-get update 
Hit:1 http://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm xenial-cm5 InRelease 
Hit:2 https://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm xenial-cm5.14.0 InRelease 
Ign:3 http://dl.google.com/linux/chrome/deb stable InRelease 
Hit:4 http://dl.google.com/linux/chrome/deb stable Release 
Hit:5 http://security.ubuntu.com/ubuntu xenial-security InRelease 
Hit:6 http://in.archive.ubuntu.com/ubuntu xenial InRelease 
Hit:8 http://in.archive.ubuntu.com/ubuntu xenial-updates InRelease 
Hit:9 http://in.archive.ubuntu.com/ubuntu xenial-backports InRelease 
Reading package lists... 
W: http://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm/dists/xenial-cm5/InRelease: Signature by key F36A89E33CC1BD0F71079007327574EE02A818DD uses weak digest algorithm (SHA1) 
W: https://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm/dists/xenial-cm5.14.0/InRelease: Signature by key F36A89E33CC1BD0F71079007327574EE02A818DD uses weak digest algorithm (SHA1) 
END (0) 
BEGIN sudo apt-get update 
Ign:1 http://dl.google.com/linux/chrome/deb stable InRelease 
Hit:2 http://dl.google.com/linux/chrome/deb stable Release 
Hit:4 http://in.archive.ubuntu.com/ubuntu xenial InRelease 
Hit:5 http://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm xenial-cm5 InRelease 
Hit:6 http://security.ubuntu.com/ubuntu xenial-security InRelease 
Hit:7 http://in.archive.ubuntu.com/ubuntu xenial-updates InRelease 
Hit:8 http://in.archive.ubuntu.com/ubuntu xenial-backports InRelease 
Hit:9 https://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm xenial-cm5.14.0 InRelease 
Reading package lists... 
W: http://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm/dists/xenial-cm5/InRelease: Signature by key F36A89E33CC1BD0F71079007327574EE02A818DD uses weak digest algorithm (SHA1) 
W: https://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm/dists/xenial-cm5.14.0/InRelease: Signature by key F36A89E33CC1BD0F71079007327574EE02A818DD uses weak digest algorithm (SHA1) 
END (0) 
Installing cloudera-manager-agent package...
BEGIN sudo dpkg -l cloudera-manager-agent | grep -E '^ii[[:space:]]*cloudera-manager-agent[[:space:]]*' 
ii cloudera-manager-agent 5.14.0-1.cm5140.p0.25~xenial-cm5 amd64 The Cloudera Manager Agent 
END (0) 
BEGIN echo oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-agent 
oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons 
END (0) 
BEGIN sudo apt-cache show cloudera-manager-agent 
Package: cloudera-manager-agent 
Source: enterprise 
Version: 5.14.0-1.cm5140.p0.25~xenial-cm5 
Architecture: amd64 
Maintainer: Cloudera Inc. <https://issues.cloudera.org> 
Installed-Size: 44066 
Depends: libc6 (>= 2.17), libcomerr2 (>= 1.01), libgcc1 (>= 1:3.0), libgssapi-krb5-2 (>= 1.10+dfsg~), libkrb5-3 (>= 1.6.dfsg.2), libssl1.0.0 (>= 1.0.2~beta3), libstdc++6 (>= 5.2), zlib1g (>= 1:1.2.0), lsb-base, psmisc, bash, libsasl2-modules, libsasl2-modules-gssapi-mit, libxslt1.1, libsqlite3-0, openssl, libssl-dev, libfuse2, fuse-utils | fuse, rpcbind, cloudera-manager-daemons (= 5.14.0-1.cm5140.p0.25~xenial-cm5), perl, python-psycopg2, python-mysqldb, apache2, iproute2 
Homepage: http://www.cloudera.com 
Priority: extra 
Section: misc 
Filename: pool/contrib/e/enterprise/cloudera-manager-agent_5.14.0-1.cm5140.p0.25~xenial-cm5_amd64.deb 
Size: 11198176 
SHA256: f5717a1581c582fdab2fde45c165ae7a9d1cfff93653b5401894e50b9a669efa 
SHA1: a043a0cd610b0846ffe7b3b4a55e4dcbea3c7c5d 
MD5sum: 7497323224c9809f6295e3936bb9dfae 
Description: The Cloudera Manager Agent 
The Agent is deployed to machines running services managed by Cloudera Manager. 
Description-md5: 482a592ee859d31443f58aa3e48fe303 

END (0) 
Version: 5.14.0-1.cm5140.p0.25~xenial-cm5 
BEGIN sudo apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-agent 
Reading package lists... 
Building dependency tree... 
Reading state information... 
cloudera-manager-agent is already the newest version (5.14.0-1.cm5140.p0.25~xenial-cm5). 
The following package was automatically installed and is no longer required: 
snap-confine 
Use 'sudo apt autoremove' to remove it. 
0 upgraded, 0 newly installed, 0 to remove and 34 not upgraded. 
END (0) 
remote package cloudera-manager-agent installed 
Installing cloudera-manager-daemons package...
BEGIN sudo dpkg -l cloudera-manager-daemons | grep -E '^ii[[:space:]]*cloudera-manager-daemons[[:space:]]*' 
ii cloudera-manager-daemons 5.14.0-1.cm5140.p0.25~xenial-cm5 all Provides daemons for monitoring Hadoop and related tools. 
END (0) 
BEGIN echo oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-daemons 
oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons 
END (0) 
BEGIN sudo apt-cache show cloudera-manager-daemons 
Package: cloudera-manager-daemons 
Source: enterprise 
Version: 5.14.0-1.cm5140.p0.25~xenial-cm5 
Architecture: all 
Maintainer: Cloudera Inc. <https://issues.cloudera.org> 
Installed-Size: 907465 
Depends: perl 
Homepage: http://www.cloudera.com 
Priority: extra 
Section: misc 
Filename: pool/contrib/e/enterprise/cloudera-manager-daemons_5.14.0-1.cm5140.p0.25~xenial-cm5_all.deb 
Size: 768061118 
SHA256: 78eeda12544d368bba1e251dd0029b1a6611f19f6246f865ebf4753308994f1e 
SHA1: 73257e96eb0db74fae4fa6b4aedb187e952e1be6 
MD5sum: 54f8a031fd25f975f521f22c31b5678e 
Description: Provides daemons for monitoring Hadoop and related tools. 
Provides daemons for monitoring Hadoop and related tools. 
Description-md5: 35bb5bc8fc2f296f791c3bda9e345340 

END (0) 
Version: 5.14.0-1.cm5140.p0.25~xenial-cm5 
BEGIN sudo apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-daemons 
Reading package lists... 
Building dependency tree... 
Reading state information... 
cloudera-manager-daemons is already the newest version (5.14.0-1.cm5140.p0.25~xenial-cm5). 
The following package was automatically installed and is no longer required: 
snap-confine 
Use 'sudo apt autoremove' to remove it. 
0 upgraded, 0 newly installed, 0 to remove and 34 not upgraded. 
END (0) 
remote package cloudera-manager-daemons installed 
Installing Unlimited Strength Encryption policy files.
Installation not requested. Step will be skipped. 
Configuring Cloudera Manager Agent...
BEGIN grep server_host=127.0.0.1 /etc/cloudera-scm-agent/config.ini 
server_host=127.0.0.1 
END (0) 
scm agent is already configured 
Starting Cloudera Manager Agent...
BEGIN /usr/sbin/service cloudera-scm-agent status 
* cloudera-scm-agent.service - LSB: Cloudera SCM Agent 
Loaded: loaded (/etc/init.d/cloudera-scm-agent; bad; vendor preset: enabled) 
Active: active (exited) since Sun 2018-01-28 10:30:14 IST; 28s ago 
Docs: man:systemd-sysv-generator(8) 
Process: 11243 ExecStop=/etc/init.d/cloudera-scm-agent stop (code=exited, status=0/SUCCESS) 
Process: 11283 ExecStart=/etc/init.d/cloudera-scm-agent start (code=exited, status=0/SUCCESS) 

Jan 28 10:30:13 chaithu systemd[1]: Starting LSB: Cloudera SCM Agent... 
Jan 28 10:30:13 chaithu su[11304]: Successful su for root by root 
Jan 28 10:30:13 chaithu su[11304]: + ??? root:root 
Jan 28 10:30:13 chaithu su[11304]: pam_unix(su:session): session opened for user root by (uid=0) 
Jan 28 10:30:14 chaithu cloudera-scm-agent[11283]: Starting cloudera-scm-agent: * cloudera-scm-agent started 
Jan 28 10:30:14 chaithu systemd[1]: Started LSB: Cloudera SCM Agent. 
END (0) 
BEGIN sudo /usr/sbin/service cloudera-scm-agent restart 
END (0) 
agent logs: 
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/' 
>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration 
>> global REDACTOR 
>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration 
>> global NOP_REDACTOR 
>>[27/Jan/2018 22:27:01 +0000] 29309 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[27/Jan/2018 22:27:01 +0000] 29309 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[27/Jan/2018 22:27:01 +0000] 29309 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[27/Jan/2018 22:32:01 +0000] 31164 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[27/Jan/2018 22:32:01 +0000] 31164 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[27/Jan/2018 22:32:01 +0000] 31164 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[27/Jan/2018 22:41:13 +0000] 1073 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[27/Jan/2018 22:41:13 +0000] 1073 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[27/Jan/2018 22:41:13 +0000] 1073 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[27/Jan/2018 22:45:01 +0000] 1327 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[27/Jan/2018 22:45:01 +0000] 1327 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[27/Jan/2018 22:45:01 +0000] 1327 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[27/Jan/2018 22:51:09 +0000] 4278 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[27/Jan/2018 22:51:09 +0000] 4278 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[27/Jan/2018 22:51:09 +0000] 4278 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 09:24:59 +0000] 1503 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 09:24:59 +0000] 1503 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 09:24:59 +0000] 1503 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 09:32:09 +0000] 6317 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 09:32:09 +0000] 6317 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 09:32:09 +0000] 6317 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 09:39:55 +0000] 1354 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 09:39:55 +0000] 1354 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 09:39:55 +0000] 1354 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 09:48:36 +0000] 6692 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 09:48:36 +0000] 6692 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 09:48:36 +0000] 6692 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 10:27:23 +0000] 9397 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 10:27:23 +0000] 9397 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 10:27:23 +0000] 9397 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 10:27:44 +0000] 10807 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 10:27:44 +0000] 10807 MainThread agent WARNING Expected mode 0751 for /run/>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration 
>> global REDACTOR 
>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration 
>> global NOP_REDACTOR 
>>[27/Jan/2018 22:27:01 +0000] 29309 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[27/Jan/2018 22:27:01 +0000] 29309 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[27/Jan/2018 22:27:01 +0000] 29309 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[27/Jan/2018 22:32:01 +0000] 31164 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[27/Jan/2018 22:32:01 +0000] 31164 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[27/Jan/2018 22:32:01 +0000] 31164 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[27/Jan/2018 22:41:13 +0000] 1073 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[27/Jan/2018 22:41:13 +0000] 1073 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[27/Jan/2018 22:41:13 +0000] 1073 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[27/Jan/2018 22:45:01 +0000] 1327 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[27/Jan/2018 22:45:01 +0000] 1327 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[27/Jan/2018 22:45:01 +0000] 1327 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[27/Jan/2018 22:51:09 +0000] 4278 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[27/Jan/2018 22:51:09 +0000] 4278 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[27/Jan/2018 22:51:09 +0000] 4278 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 09:24:59 +0000] 1503 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 09:24:59 +0000] 1503 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 09:24:59 +0000] 1503 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 09:32:09 +0000] 6317 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 09:32:09 +0000] 6317 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 09:32:09 +0000] 6317 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 09:39:55 +0000] 1354 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 09:39:55 +0000] 1354 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 09:39:55 +0000] 1354 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 09:48:36 +0000] 6692 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 09:48:36 +0000] 6692 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 09:48:36 +0000] 6692 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 10:27:23 +0000] 9397 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 10:27:23 +0000] 9397 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 10:27:23 +0000] 9397 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 10:27:44 +0000] 10807 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 10:27:44 +0000] 10807 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 10:27:44 +0000] 10807 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 10:30:14 +0000] 11316 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 10:30:14 +0000] 11316 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 10:30:14 +0000] 11316 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 10:30:44 +0000] 12757 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 10:30:44 +0000] 12757 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 10:30:44 +0000] 12757 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
cloudera-scm-agent but was 0755 
>>[28/Jan/2018 10:27:44 +0000] 10807 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 10:30:14 +0000] 11316 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 10:30:14 +0000] 11316 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 10:30:14 +0000] 11316 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 10:30:44 +0000] 12757 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 10:30:44 +0000] 12757 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 
>>[28/Jan/2018 10:30:44 +0000] 12757 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent 
END (0) 
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/' 
>>[28/Jan/2018 10:30:18 +0000] 11330 MainThread agent INFO Flood daemon (re)start attempt 
>>[28/Jan/2018 10:30:19 +0000] 11330 MainThread agent INFO Triggering supervisord update. 
>>[28/Jan/2018 10:30:20 +0000] 11330 MainThread agent INFO Active parcel list updated; recalculating component info. 
>>[28/Jan/2018 10:30:20 +0000] 11330 MainThread throttling_logger INFO Identified java component java8 with full version JAVA_HOME=/usr/lib/jvm/default-java openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode) for requested version misc. 
>>[28/Jan/2018 10:30:20 +0000] 11330 MainThread throttling_logger INFO Identified java component java7 with full version JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) for requested version 7. 
>>[28/Jan/2018 10:30:21 +0000] 11330 MainThread throttling_logger INFO Identified java component java8 with full version JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode) for requested version 8. 
>>[28/Jan/2018 10:30:21 +0000] 11330 Monitor-HostMonitor throttling_logger ERROR Failed to collect NTP metrics 
>>Traceback (most recent call last): 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 48, in collect 
>> self.collect_ntpd() 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 66, in collect_ntpd 
>> result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout) 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 38, in _subprocess_with_timeout 
>> return subprocess_with_timeout(args, timeout) 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/subprocess_timeout.py", line 49, in subprocess_with_timeout 
>> p = subprocess.Popen(**kwargs) 
>> File "/usr/lib/python2.7/subprocess.py", line 711, in __init__ 
>> errread, errwrite) 
>> File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child 
>> raise child_exception 
>>OSError: [Errno 2] No such file or directory 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread agent INFO Stopping agent... 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread agent INFO No extant cgroups; unmounting any cgroup roots 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread agent INFO 2 processes are being managed; Supervisor will continue to run. 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus STOPPING 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('chaithu', 9000)) shut down 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Stopped thread '_TimeoutMonitor'. 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus STOPPED 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus STOPPING 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('chaithu', 9000)) already shut down 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE No thread running for None. 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus STOPPED 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus EXITING 
>>>[28/Jan/2018 10:30:18 +0000] 11330 MainThread agent INFO Flood daemon (re)start attempt 
>>[28/Jan/2018 10:30:19 +0000] 11330 MainThread agent INFO Triggering supervisord update. 
>>[28/Jan/2018 10:30:20 +0000] 11330 MainThread agent INFO Active parcel list updated; recalculating component info. 
>>[28/Jan/2018 10:30:20 +0000] 11330 MainThread throttling_logger INFO Identified java component java8 with full version JAVA_HOME=/usr/lib/jvm/default-java openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode) for requested version misc. 
>>[28/Jan/2018 10:30:20 +0000] 11330 MainThread throttling_logger INFO Identified java component java7 with full version JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) for requested version 7. 
>>[28/Jan/2018 10:30:21 +0000] 11330 MainThread throttling_logger INFO Identified java component java8 with full version JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode) for requested version 8. 
>>[28/Jan/2018 10:30:21 +0000] 11330 Monitor-HostMonitor throttling_logger ERROR Failed to collect NTP metrics 
>>Traceback (most recent call last): 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 48, in collect 
>> self.collect_ntpd() 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 66, in collect_ntpd 
>> result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout) 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 38, in _subprocess_with_timeout 
>> return subprocess_with_timeout(args, timeout) 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/subprocess_timeout.py", line 49, in subprocess_with_timeout 
>> p = subprocess.Popen(**kwargs) 
>> File "/usr/lib/python2.7/subprocess.py", line 711, in __init__ 
>> errread, errwrite) 
>> File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child 
>> raise child_exception 
>>OSError: [Errno 2] No such file or directory 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread agent INFO Stopping agent... 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread agent INFO No extant cgroups; unmounting any cgroup roots 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread agent INFO 2 processes are being managed; Supervisor will continue to run. 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus STOPPING 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('chaithu', 9000)) shut down 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Stopped thread '_TimeoutMonitor'. 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus STOPPED 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus STOPPING 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('chaithu', 9000)) already shut down 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE No thread running for None. 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus STOPPED 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus EXITING 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus EXITED 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread agent INFO Agent exiting; caught signal 15 
>>[28/Jan/2018 10:30:43 +0000] 11330 Dummy-14 daemonize WARNING Stopping daemon. 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread __init__ INFO Agent UUID file was last modified at 2018-01-27 22:27:01.470698 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO ================================================================================ 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Agent Protocol Version: 4 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Using Host ID: 071a1f6d-8ac5-40ae-bd37-fa72ae865fc0 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Using directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Using supervisor binary path: /usr/lib/cmf/agent/build/env/bin/supervisord 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation. 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Agent Logging Level: INFO 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO No command line vars 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Found database jar: /usr/share/java/mysql-connector-java.jar 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type) 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Agent starting as pid 12771 user root(0) group root(0). 
>[28/Jan/2018 10:30:43 +0000] 11330 MainThread _cplogging INFO [28/Jan/2018:10:30:43] ENGINE Bus EXITED 
>>[28/Jan/2018 10:30:43 +0000] 11330 MainThread agent INFO Agent exiting; caught signal 15 
>>[28/Jan/2018 10:30:43 +0000] 11330 Dummy-14 daemonize WARNING Stopping daemon. 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread __init__ INFO Agent UUID file was last modified at 2018-01-27 22:27:01.470698 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO ================================================================================ 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO SCM Agent Version: 5.14.0 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Agent Protocol Version: 4 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Using Host ID: 071a1f6d-8ac5-40ae-bd37-fa72ae865fc0 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Using directory: /run/cloudera-scm-agent 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Using supervisor binary path: /usr/lib/cmf/agent/build/env/bin/supervisord 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation. 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Agent Logging Level: INFO 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO No command line vars 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Found database jar: /usr/share/java/mysql-connector-java.jar 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type) 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar 
>>[28/Jan/2018 10:30:44 +0000] 12771 MainThread agent INFO Agent starting as pid 12771 user root(0) group root(0). 
END (0) 
end of agent logs. 
scm agent restarted 
Installation script completed successfully.
all done 
closing logging file descriptor 

Below are the local linux details:

 

 
chaithu@chaithu:~$ hostname
chaithu
chaithu@chaithu:~$ hostname -f
chaithu
chaithu@chaithu:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 chaithu

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
chaithu@chaithu:~$ sudo chkconfig iptables --list
sudo: chkconfig: command not found
chaithu@chaithu:~$ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:7180            0.0.0.0:*               LISTEN      1353/java       
tcp        0      0 0.0.0.0:7182            0.0.0.0:*               LISTEN      1353/java       
tcp        0      0 0.0.0.0:4433            0.0.0.0:*               LISTEN      12930/python2.7 
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN      1162/dnsmasq    
tcp        0      0 127.0.0.1:7190          0.0.0.0:*               LISTEN      12930/python2.7 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1031/sshd       
tcp        0      0 0.0.0.0:7191            0.0.0.0:*               LISTEN      12930/python2.7 
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      5112/cupsd      
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      1077/postgres   
tcp        0      0 127.0.0.1:19001         0.0.0.0:*               LISTEN      1632/python     
tcp        0      0 127.0.1.1:9000          0.0.0.0:*               LISTEN      12771/python2.7 
tcp        0      0 0.0.0.0:7432            0.0.0.0:*               LISTEN      1434/postgres   
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1045/mysqld     
tcp6       0      0 :::80                   :::*                    LISTEN      1441/apache2    
tcp6       0      0 :::4434                 :::*                    LISTEN      12930/python2.7 
tcp6       0      0 :::22                   :::*                    LISTEN      1031/sshd       
tcp6       0      0 :::7191                 :::*                    LISTEN      12930/python2.7 
tcp6       0      0 ::1:631                 :::*                    LISTEN      5112/cupsd      
tcp6       0      0 :::7432                 :::*                    LISTEN      1434/postgres   
udp        0      0 0.0.0.0:57656           0.0.0.0:*                           902/avahi-daemon: r
udp        0      0 0.0.0.0:631             0.0.0.0:*                           5113/cups-browsed
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           2703/chrome     
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           902/avahi-daemon: r
udp        0      0 0.0.0.0:39330           0.0.0.0:*                           1162/dnsmasq    
udp        0      0 0.0.0.0:7191            0.0.0.0:*                           12930/python2.7 
udp        0      0 127.0.1.1:53            0.0.0.0:*                           1162/dnsmasq    
udp        0      0 0.0.0.0:68              0.0.0.0:*                           1117/dhclient   
udp6       0      0 :::5353                 :::*                                2703/chrome     
udp6       0      0 :::5353                 :::*                                902/avahi-daemon: r
udp6       0      0 :::46382                :::*                                902/avahi-daemon: r
udp6       0      0 :::7191                 :::*                                12930/python2.7 
Kindly help me to fix this issue and let me proceed further. Thanks in advance. Do let me know if any more info required.
1 REPLY 1

Super Guru

@Chaithu,

 

The behavior you describe indicates that the installation went well but then the agent could not heartbeat to Cloudera Manager as expected.  Please check the agent log for clues.

By default, the file is /var/log/cloudera-scm-agent/cloudera-scm-agent.log