Support Questions

Find answers, ask questions, and share your expertise

Cloudera Manager. Agent cannot connect with supervisor. Failed to receive heartbeat from agent

avatar
Explorer

Hello all,

 

I've been searching in the forum for this error. I know this is a very common error and I've read a lot of differents posts with related information and solutions. But I still can't solve the problem. I've been trying for two days, and maybe you can help me with something I'm passing through.

 

The point is that I'm trying to install a 3 nodes cluster with Cloudera Manager in Ubuntu 14.04 LTS using the Installation Path A. In the Cluster installation I receive the following error:

 

 

 Installation failed. Failed to receive heartbeat from agent.

    Ensure that the host's hostname is configured properly.
    Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
    Ensure that ports 9000 and 9001 are not in use on the host being added.
    Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).
    If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.

 

All ports are open. Firewall and SeLinux are deactivated and the following commands with their results are:

hostname

vbigdata0srv.domain.com

 

hostname -f

vbigdata0srv.domain.com

 

python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'

vbigdata0srv.domain.com 192.168.XXX.XXX (The right IP)

 

*This is the same for  all 3 nodes (vbigdata1srv and vbigdata2srv) with their respective IPs.

 

 

 

The log looks like this, showing as error: ">>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent ERROR Failed to connect to previous supervisor. "

 

 

/tmp/scm_prepare_node.25wJDbEq
using SSH_CLIENT to get the SCM hostname: 192.168.XXX.XXX 38281 22
opening logging file descriptor
Starting installation script...
Acquiring installation lock...
BEGIN flock 4
END (0)
Detecting root privileges...
effective UID is 1200
BEGIN which pbrun
END (1)
BEGIN sudo -S id
uid=0(root) gid=0(root) groups=0(root)
END (0)
Using 'sudo ' to acquire root privileges
Detecting distribution...
BEGIN grep 'Ubuntu' /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_DESCRIPTION="Ubuntu 14.04 LTS"
END (0)
BEGIN grep DISTRIB_CODENAME /etc/lsb-release
DISTRIB_CODENAME=trusty
END (0)
BEGIN echo DISTRIB_CODENAME=trusty | cut -d = -f 2
trusty
END (0)
Detecting Cloudera Manager Server...
BEGIN host -t PTR 192.168.XXX.XXX
Host XXX.XXX.168.192.in-addr.arpa. not found: 3(NXDOMAIN)
END (1)
BEGIN which python
END (0)
BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' 192.168.XXX.XXX 7182
/usr/bin/python
END (0)
BEGIN which wget
END (0)
/usr/bin/wget
BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo
END (4)
Installing package repositories...
validating format of repository file /tmp/scm_prepare_node.25wJDbEq/repos/ubuntu_trusty/cloudera-manager.list
installing repository file /tmp/scm_prepare_node.25wJDbEq/repos/ubuntu_trusty/cloudera-manager.list
repository file /tmp/scm_prepare_node.25wJDbEq/repos/ubuntu_trusty/cloudera-manager.list installed
BEGIN sudo apt-key add /tmp/scm_prepare_node.25wJDbEq/archive.key
OK
END (0)
installing priority file /tmp/scm_prepare_node.25wJDbEq/priorities/ubuntu_trusty/cloudera.pref
priority file /tmp/scm_prepare_node.25wJDbEq/priorities/ubuntu_trusty/cloudera.pref installed
Refreshing package metadata...
BEGIN sudo apt-get update
Hit http://ppa.launchpad.net trusty InRelease
Ign http://es.archive.ubuntu.com trusty InRelease
Hit http://security.ubuntu.com trusty-security InRelease
Hit http://archive.cloudera.com trusty-cm5 InRelease
Ign http://extras.ubuntu.com trusty InRelease
Hit http://es.archive.ubuntu.com trusty-updates InRelease
Hit http://ppa.launchpad.net trusty/main amd64 Packages
Hit http://archive.cloudera.com trusty-cm5/contrib Sources
Hit http://es.archive.ubuntu.com trusty-backports InRelease
Hit http://extras.ubuntu.com trusty Release.gpg
Hit http://security.ubuntu.com trusty-security/main Sources
Hit http://archive.cloudera.com trusty-cm5/contrib amd64 Packages
Hit http://ppa.launchpad.net trusty/main i386 Packages
Hit http://es.archive.ubuntu.com trusty Release.gpg
Hit http://security.ubuntu.com trusty-security/restricted Sources
Hit http://extras.ubuntu.com trusty Release
Hit http://ppa.launchpad.net trusty/main Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/main Sources
Hit http://security.ubuntu.com trusty-security/universe Sources
Hit http://es.archive.ubuntu.com trusty-updates/restricted Sources
Hit http://extras.ubuntu.com trusty/main Sources
Hit http://security.ubuntu.com trusty-security/multiverse Sources
Hit https://archive.cloudera.com trusty-cm5.12.0 InRelease
Hit http://es.archive.ubuntu.com trusty-updates/universe Sources
Hit https://archive.cloudera.com trusty-cm5.12.0/contrib amd64 Packages
Hit http://extras.ubuntu.com trusty/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en_US
Get:1 https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US
Hit http://es.archive.ubuntu.com trusty-updates/multiverse Sources
Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en
Hit http://security.ubuntu.com trusty-security/restricted amd64 Packages
Hit http://extras.ubuntu.com trusty/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/restricted amd64 Packages
Hit http://security.ubuntu.com trusty-security/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/universe amd64 Packages
Hit http://security.ubuntu.com trusty-security/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/main i386 Packages
Hit http://security.ubuntu.com trusty-security/restricted i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/restricted i386 Packages
Hit http://security.ubuntu.com trusty-security/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/universe i386 Packages
Hit http://security.ubuntu.com trusty-security/multiverse i386 Packages
Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US
Hit http://es.archive.ubuntu.com trusty-updates/multiverse i386 Packages
Hit http://security.ubuntu.com trusty-security/main Translation-en
Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://security.ubuntu.com trusty-security/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/multiverse Translation-en
Hit http://security.ubuntu.com trusty-security/restricted Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/restricted Translation-en
Ign http://extras.ubuntu.com trusty/main Translation-en_US
Hit http://es.archive.ubuntu.com trusty-updates/universe Translation-en
Ign http://extras.ubuntu.com trusty/main Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/main Sources
Hit http://es.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://es.archive.ubuntu.com trusty-backports/universe Sources
Hit http://es.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://es.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/restricted i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/multiverse i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/main Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/restricted Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/universe Translation-en
Hit http://es.archive.ubuntu.com trusty Release
Hit http://es.archive.ubuntu.com trusty/main Sources
Hit http://es.archive.ubuntu.com trusty/restricted Sources
Hit http://es.archive.ubuntu.com trusty/universe Sources
Hit http://es.archive.ubuntu.com trusty/multiverse Sources
Hit http://es.archive.ubuntu.com trusty/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://es.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://es.archive.ubuntu.com trusty/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty/main i386 Packages
Hit http://es.archive.ubuntu.com trusty/restricted i386 Packages
Hit http://es.archive.ubuntu.com trusty/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty/multiverse i386 Packages
Hit http://es.archive.ubuntu.com trusty/main Translation-en
Hit http://es.archive.ubuntu.com trusty/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty/restricted Translation-en
Hit http://es.archive.ubuntu.com trusty/universe Translation-en
Ign http://es.archive.ubuntu.com trusty/main Translation-en_US
Ign http://es.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://es.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://es.archive.ubuntu.com trusty/universe Translation-en_US
Reading package lists...
END (0)
BEGIN sudo apt-get update
Hit http://archive.cloudera.com trusty-cm5 InRelease
Hit http://archive.cloudera.com trusty-cm5/contrib Sources
Hit http://ppa.launchpad.net trusty InRelease
Ign http://es.archive.ubuntu.com trusty InRelease
Ign http://extras.ubuntu.com trusty InRelease
Hit http://archive.cloudera.com trusty-cm5/contrib amd64 Packages
Hit http://ppa.launchpad.net trusty/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates InRelease
Hit http://extras.ubuntu.com trusty Release.gpg
Hit http://ppa.launchpad.net trusty/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports InRelease
Hit http://extras.ubuntu.com trusty Release
Hit http://ppa.launchpad.net trusty/main Translation-en
Hit http://es.archive.ubuntu.com trusty Release.gpg
Hit https://archive.cloudera.com trusty-cm5.12.0 InRelease
Hit http://extras.ubuntu.com trusty/main Sources
Hit http://es.archive.ubuntu.com trusty-updates/main Sources
Hit https://archive.cloudera.com trusty-cm5.12.0/contrib amd64 Packages
Hit http://security.ubuntu.com trusty-security InRelease
Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en_US
Get:1 https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US
Hit http://extras.ubuntu.com trusty/main amd64 Packages
Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/restricted Sources
Hit http://extras.ubuntu.com trusty/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/universe Sources
Hit http://es.archive.ubuntu.com trusty-updates/multiverse Sources
Hit http://security.ubuntu.com trusty-security/main Sources
Hit http://es.archive.ubuntu.com trusty-updates/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/restricted amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/universe amd64 Packages
Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US
Hit http://security.ubuntu.com trusty-security/restricted Sources
Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/restricted i386 Packages
Hit http://security.ubuntu.com trusty-security/universe Sources
Hit http://es.archive.ubuntu.com trusty-updates/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/multiverse i386 Packages
Ign http://extras.ubuntu.com trusty/main Translation-en_US
Hit http://es.archive.ubuntu.com trusty-updates/main Translation-en
Ign http://extras.ubuntu.com trusty/main Translation-en
Hit http://security.ubuntu.com trusty-security/multiverse Sources
Hit http://es.archive.ubuntu.com trusty-updates/multiverse Translation-en
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/restricted Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/universe Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/main Sources
Hit http://security.ubuntu.com trusty-security/restricted amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://es.archive.ubuntu.com trusty-backports/universe Sources
Hit http://es.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://security.ubuntu.com trusty-security/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/restricted i386 Packages
Hit http://security.ubuntu.com trusty-security/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/multiverse i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/main Translation-en
Hit http://security.ubuntu.com trusty-security/restricted i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/restricted Translation-en
Hit http://security.ubuntu.com trusty-security/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/universe Translation-en
Hit http://es.archive.ubuntu.com trusty Release
Hit http://security.ubuntu.com trusty-security/multiverse i386 Packages
Hit http://es.archive.ubuntu.com trusty/main Sources
Hit http://es.archive.ubuntu.com trusty/restricted Sources
Hit http://es.archive.ubuntu.com trusty/universe Sources
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://es.archive.ubuntu.com trusty/multiverse Sources
Hit http://es.archive.ubuntu.com trusty/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://security.ubuntu.com trusty-security/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://es.archive.ubuntu.com trusty/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty/main i386 Packages
Hit http://security.ubuntu.com trusty-security/restricted Translation-en
Hit http://es.archive.ubuntu.com trusty/restricted i386 Packages
Hit http://es.archive.ubuntu.com trusty/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty/multiverse i386 Packages
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Hit http://es.archive.ubuntu.com trusty/main Translation-en
Hit http://es.archive.ubuntu.com trusty/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty/restricted Translation-en
Hit http://es.archive.ubuntu.com trusty/universe Translation-en
Ign http://es.archive.ubuntu.com trusty/main Translation-en_US
Ign http://es.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://es.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://es.archive.ubuntu.com trusty/universe Translation-en_US
Reading package lists...
END (0)
Installing cloudera-manager-agent package...
BEGIN sudo dpkg -l cloudera-manager-agent | grep -E '^ii[[:space:]]*cloudera-manager-agent[[:space:]]*'
ii cloudera-manager-agent 5.12.0-1.cm5120.p0.120~trusty-cm5 amd64 The Cloudera Manager Agent
END (0)
BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-agent
END (0)
oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons
BEGIN sudo apt-cache show cloudera-manager-agent
Package: cloudera-manager-agent
Source: enterprise
Version: 5.12.0-1.cm5120.p0.120~trusty-cm5
Architecture: amd64
Maintainer: Cloudera Inc. <https://issues.cloudera.org>
Installed-Size: 41468
Depends: libc6 (>= 2.17), libcomerr2 (>= 1.01), libgcc1 (>= 1:4.1.1), libgssapi-krb5-2 (>= 1.10+dfsg~), libkrb5-3 (>= 1.6.dfsg.2), libssl1.0.0 (>= 1.0.1), libstdc++6 (>= 4.6), zlib1g (>= 1:1.2.0), lsb-base, psmisc, bash, libsasl2-modules, libsasl2-modules-gssapi-mit, libxslt1.1, libsqlite3-0, openssl, libssl-dev, libfuse2, fuse-utils | fuse, rpcbind, cloudera-manager-daemons (= 5.12.0-1.cm5120.p0.120~trusty-cm5), perl, python-psycopg2, python-mysqldb, apache2, iproute2
Homepage: http://www.cloudera.com
Priority: extra
Section: misc
Filename: pool/contrib/e/enterprise/cloudera-manager-agent_5.12.0-1.cm5120.p0.120~trusty-cm5_amd64.deb
Size: 9757280
SHA256: c5204a5ae2e3a21cb1c34e7b4472d5a4e7f074d53cdeab9d1a2a28b2504f8d7d
SHA1: cf7d444edcaef6144c63c3c7376414942c047c0e
MD5sum: b47a11f7c8221f4e1819e51c44e347fe
Description: The Cloudera Manager Agent
The Agent is deployed to machines running services managed by Cloudera Manager.
Description-md5: 482a592ee859d31443f58aa3e48fe303

END (0)
Version: 5.12.0-1.cm5120.p0.120~trusty-cm5
BEGIN sudo apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-agent
Reading package lists...
Building dependency tree...
Reading state information...
cloudera-manager-agent is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 584 not upgraded.
END (0)
remote package cloudera-manager-agent installed
Installing cloudera-manager-daemons package...
BEGIN sudo dpkg -l cloudera-manager-daemons | grep -E '^ii[[:space:]]*cloudera-manager-daemons[[:space:]]*'
ii cloudera-manager-daemons 5.12.0-1.cm5120.p0.120~trusty-cm5 all Provides daemons for monitoring Hadoop and related tools.
END (0)
BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-daemons
oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons
END (0)
BEGIN sudo apt-cache show cloudera-manager-daemons
Package: cloudera-manager-daemons
Source: enterprise
Version: 5.12.0-1.cm5120.p0.120~trusty-cm5
Architecture: all
Maintainer: Cloudera Inc. <https://issues.cloudera.org>
Installed-Size: 870556
Depends: perl
Homepage: http://www.cloudera.com
Priority: extra
Section: misc
Filename: pool/contrib/e/enterprise/cloudera-manager-daemons_5.12.0-1.cm5120.p0.120~trusty-cm5_all.deb
Size: 743034520
SHA256: 2098af13c70588b765e759b613948a52673fb3cc8ac688d912b8315099361ce4
SHA1: d59c7c1d7df42592707011922a2c27554b03f7eb
MD5sum: ec945225a977da256e85a3ae28e54766
Description: Provides daemons for monitoring Hadoop and related tools.
Provides daemons for monitoring Hadoop and related tools.
Description-md5: 35bb5bc8fc2f296f791c3bda9e345340

END (0)
Version: 5.12.0-1.cm5120.p0.120~trusty-cm5
BEGIN sudo apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-daemons
Reading package lists...
Building dependency tree...
Reading state information...
cloudera-manager-daemons is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 584 not upgraded.
END (0)
remote package cloudera-manager-daemons installed
Installing Unlimited Strength Encryption policy files.
Installation not requested. Step will be skipped.
Configuring Cloudera Manager Agent...
BEGIN grep server_host=192.168.XXX.XXX /etc/cloudera-scm-agent/config.ini
server_host=192.168.XXX.XXX
END (0)
scm agent is already configured
Starting Cloudera Manager Agent...
BEGIN /usr/sbin/service cloudera-scm-agent status
Checking for service cloudera-scm-agent * cloudera-scm-agent is not running
END (3)
BEGIN sudo /usr/sbin/service cloudera-scm-agent start
Starting cloudera-scm-agent: bash: /var/log/cloudera-scm-agent/cloudera-scm-agent.out: Permission denied
* Couldn't start cloudera-scm-agent

END (0)
agent logs:
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/'
>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration
>> global REDACTOR
>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration
>> global NOP_REDACTOR
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration
>> global REDACTOR
>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration
>> global NOP_REDACTOR
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
END (0)
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/'
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Found cgroups subsystem: cpuacct
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Found cgroups subsystem: memory
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Found cgroups subsystem: blkio
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/memory
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpu
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpuacct
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/blkio
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': -1, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True}
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Setting up supervisord event monitor.
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Using timeout of 2.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Using join timeout of 0.100000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Using tolerance of 60.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4', 'xfs']
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread kt_renewer INFO Agent wide credential cache set to /run/cloudera-scm-agent/krb5cc_cm_agent_0
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Using max_collection_wait_seconds of 10.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/monitor/tasktracker/schema.json
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread ntp_monitor INFO Using timeout of 2.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread dns_names INFO Using timeout of 30.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread __init__ INFO Created DNS monitor.
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread __init__ INFO Importing metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/monitor/schema.json
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'ORACLE_HOME': '/usr/share/oracle/instantclient', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/process
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/flood
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent        ERROR    Failed to connect to previous supervisor.
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2109, in find_or_start_supervisor
>> self.configure_supervisor_clients()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2290, in configure_supervisor_clients
>> supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")])
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 1599, in realize
>> Options.realize(self, *arg, **kw)
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 333, in realize
>> self.process_config()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 341, in process_config
>> self.process_config_file(do_usage)
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 376, in process_config_file
>> self.usage(str(msg))
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 164, in usage
>> self.exit(2)
>>SystemExit: 2
>>[07/Aug/2017 10:25:17 +0000] 10429 Dummy-1 daemonize WARNING Stopping daemon.
>>[07/Aug/2017 10:25:17 +0000] 10429 Dummy-1 agent INFO Stopping agent...
>>[07/Aug/2017 10:25:17 +0000] 10429 Dummy-1 agent INFO No extant cgroups; unmounting any cgroup roots
END (0)
end of agent logs.
scm agent started 

 

I tried reinstalling and so on, but I can't solve this. This is second time I try to install Cloudera Manager. First one was in a single node and it went all right at first time.


Anyone have any idea on how to solve this?


Thanks you so much in advance.

 

1 ACCEPTED SOLUTION

avatar
Master Collaborator

If running as sudo the results is giving you 

 

OSError: [Errno 13] Permission denied: '/run/user/112/gvfs'

You will need to temporary unmount that tmpfs mount/path, which will allow the Cloudera Manager Agent to start - you are hitting a known issue which will be addressed in the future maintenance release.

 

Regards,

 

Michalis

View solution in original post

14 REPLIES 14

avatar
Explorer

thus,i have post a new topic at https://community.cloudera.com/t5/Cloudera-Manager-Installation/CDH-5-12-0-clouder-manager-agent-can... infomation is post on this url.thks for help.

avatar
Champion

if you have the same error , then the suggest work around is to unmount the "tmpfs " 

To know the location of tmpfs path check your /etc/fstab file you will see the mount point path . 

 

avatar
Explorer

[root@bigdata-master cloudera-scm-agent]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Tue Aug 8 03:48:20 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=6a383e2f-6ebf-4654-b3ef-cd7917564a78 / xfs defaults 0 0
UUID=b347299f-6786-4f12-a3eb-84d580423ebc /boot xfs defaults 0 0
UUID=6523647e-3167-4caf-a3cc-135dc081b1b8 swap swap defaults 0 0

 

this is my fstab file content,i am not sure where the problem is.

avatar
Explorer

Hi,

 

I am facing the same issue where, cloudera agent is exiting because it is unable to find the supervisord.conf file. The conf file is not getting created. 

 

Error see in cloudera-scm-agent.log file

 

[24/May/2018 14:19:31 +0000] 10889 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
/supervisor/include
[24/May/2018 14:19:31 +0000] 10889 MainThread agent ERROR Failed to connect to previous supervisor.
 Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2110, in find_or_start_supervisor
self.get_supervisor_process_info()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2254, in get_supervisor_process_info
self.identifier = self.supervisor_client.supervisor.getIdentification()
File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request
self.connection.request('POST', handler, request_body, self.headers)
File "/usr/lib64/python2.7/httplib.py", line 1017, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.7/httplib.py", line 1051, in _send_request
self.endheaders(body)
File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders
self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output
self.send(msg)
File "/usr/lib64/python2.7/httplib.py", line 826, in send
self.connect()
File "/usr/lib64/python2.7/httplib.py", line 807, in connect
self.timeout, self.source_address)
File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
raise err

error: [Errno 111] Connection refused
[24/May/2018 14:19:31 +0000] 10889 Dummy-1 daemonize WARNING Stopping daemon.
[24/May/2018 14:19:31 +0000] 10889 Dummy-1 agent INFO Stopping agent...
[24/May/2018 14:19:31 +0000] 10889 Dummy-1 agent INFO No extant cgroups; unmounting any cgroup roots

 

The cloudera manager and agent version is 5.12.0

 

We did have a server failure and after the reboot of the server, the agent does not start.

 

Ran the below the command:

/usr/lib64/cmf/agent/build/env/bin/python /usr/lib64/cmf/agent/build/env/bin/cmf-agent --package_dir /usr/lib64/cmf/service --agent_dir /var/run/cloudera-scm-agent --lib_dir /var/lib/cloudera-scm-agent --logfile /var/log/cloudera-scm-agent/cloudera-scm-agent.log --comm_name cmf-agent --pidfile /var/run/cloudera-scm-agent/cloudera-scm-agent.pid

 

Output:


[24/May/2018 14:27:45 +0000] 11274 MainThread agent INFO SCM Agent Version: 5.12.0
[24/May/2018 14:27:45 +0000] 11274 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
[24/May/2018 14:27:45 +0000] 11274 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
[24/May/2018 14:27:45 +0000] 11274 MainThread agent INFO Not starting a new session.
[24/May/2018 14:27:45 +0000] 11274 MainThread agent INFO Adding env vars that start with CMF_AGENT_
[24/May/2018 14:27:45 +0000] 11274 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/bin/cmf-agent", line 12, in <module>
load_entry_point('cmf==5.12.0', 'console_scripts', 'cmf-agent')()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 3095, in main
main_impl()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 3078, in main_impl
agent.start()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 804, in start
self.find_or_start_supervisor()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2151, in find_or_start_supervisor
if not mount_tmpfs(process_dir, self.args.clear_agent_dir, self.sudo):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/tmpfs.py", line 62, in mount_tmpfs
if os.path.samefile(p.mountpoint, path) and p.fstype == "tmpfs" and "noexec" not in p.opts:
File "/usr/lib64/cmf/agent/build/env/lib64/python2.7/posixpath.py", line 162, in samefile
s1 = os.stat(f1)
OSError: [Errno 2] No such file or directory: 'net:[4026532100]'

 

Please may you help on this? Is the issue fixed in v5.12.2?

avatar
New Contributor

To resolve this issue you need to check first all port opened on your server service listing to the port no, use command - sudo netstat -lpten check if any thing is running on 9000 or 90001, mostly java services required for set up is running on port 9000, and cloudera-scm-agent listner also runs on port 9000. to over come this issue you can re-configure theports as well in /etc/cloudera-scm-agent/config.ini by changing as below:-

 

# Hostname of the CM server.
server_host=10.34.156.118

# Port that the CM server is listening on.
server_port=7182

## It should not normally be necessary to modify these.
# Port that the CM agent should listen on.
listening_port=9001

 

It should not normally be necessary to modify these. # Port that the CM agent should listen on 9001

and then restart the cloudera-scm-agent service by command:- $ service cloudera-scm-agent restart TO verify this port is not activated for other sshd service check Ports opened in /etc/ssh/sshd_config.

 

I hope this resolution will work for others too.

Cheers,

Ankit Gupta