Support Questions

Find answers, ask questions, and share your expertise

Cloudera Manager. Agent cannot connect with supervisor. Failed to receive heartbeat from agent

avatar
Explorer

Hello all,

 

I've been searching in the forum for this error. I know this is a very common error and I've read a lot of differents posts with related information and solutions. But I still can't solve the problem. I've been trying for two days, and maybe you can help me with something I'm passing through.

 

The point is that I'm trying to install a 3 nodes cluster with Cloudera Manager in Ubuntu 14.04 LTS using the Installation Path A. In the Cluster installation I receive the following error:

 

 

 Installation failed. Failed to receive heartbeat from agent.

    Ensure that the host's hostname is configured properly.
    Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
    Ensure that ports 9000 and 9001 are not in use on the host being added.
    Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).
    If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.

 

All ports are open. Firewall and SeLinux are deactivated and the following commands with their results are:

hostname

vbigdata0srv.domain.com

 

hostname -f

vbigdata0srv.domain.com

 

python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'

vbigdata0srv.domain.com 192.168.XXX.XXX (The right IP)

 

*This is the same for  all 3 nodes (vbigdata1srv and vbigdata2srv) with their respective IPs.

 

 

 

The log looks like this, showing as error: ">>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent ERROR Failed to connect to previous supervisor. "

 

 

/tmp/scm_prepare_node.25wJDbEq
using SSH_CLIENT to get the SCM hostname: 192.168.XXX.XXX 38281 22
opening logging file descriptor
Starting installation script...
Acquiring installation lock...
BEGIN flock 4
END (0)
Detecting root privileges...
effective UID is 1200
BEGIN which pbrun
END (1)
BEGIN sudo -S id
uid=0(root) gid=0(root) groups=0(root)
END (0)
Using 'sudo ' to acquire root privileges
Detecting distribution...
BEGIN grep 'Ubuntu' /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_DESCRIPTION="Ubuntu 14.04 LTS"
END (0)
BEGIN grep DISTRIB_CODENAME /etc/lsb-release
DISTRIB_CODENAME=trusty
END (0)
BEGIN echo DISTRIB_CODENAME=trusty | cut -d = -f 2
trusty
END (0)
Detecting Cloudera Manager Server...
BEGIN host -t PTR 192.168.XXX.XXX
Host XXX.XXX.168.192.in-addr.arpa. not found: 3(NXDOMAIN)
END (1)
BEGIN which python
END (0)
BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' 192.168.XXX.XXX 7182
/usr/bin/python
END (0)
BEGIN which wget
END (0)
/usr/bin/wget
BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo
END (4)
Installing package repositories...
validating format of repository file /tmp/scm_prepare_node.25wJDbEq/repos/ubuntu_trusty/cloudera-manager.list
installing repository file /tmp/scm_prepare_node.25wJDbEq/repos/ubuntu_trusty/cloudera-manager.list
repository file /tmp/scm_prepare_node.25wJDbEq/repos/ubuntu_trusty/cloudera-manager.list installed
BEGIN sudo apt-key add /tmp/scm_prepare_node.25wJDbEq/archive.key
OK
END (0)
installing priority file /tmp/scm_prepare_node.25wJDbEq/priorities/ubuntu_trusty/cloudera.pref
priority file /tmp/scm_prepare_node.25wJDbEq/priorities/ubuntu_trusty/cloudera.pref installed
Refreshing package metadata...
BEGIN sudo apt-get update
Hit http://ppa.launchpad.net trusty InRelease
Ign http://es.archive.ubuntu.com trusty InRelease
Hit http://security.ubuntu.com trusty-security InRelease
Hit http://archive.cloudera.com trusty-cm5 InRelease
Ign http://extras.ubuntu.com trusty InRelease
Hit http://es.archive.ubuntu.com trusty-updates InRelease
Hit http://ppa.launchpad.net trusty/main amd64 Packages
Hit http://archive.cloudera.com trusty-cm5/contrib Sources
Hit http://es.archive.ubuntu.com trusty-backports InRelease
Hit http://extras.ubuntu.com trusty Release.gpg
Hit http://security.ubuntu.com trusty-security/main Sources
Hit http://archive.cloudera.com trusty-cm5/contrib amd64 Packages
Hit http://ppa.launchpad.net trusty/main i386 Packages
Hit http://es.archive.ubuntu.com trusty Release.gpg
Hit http://security.ubuntu.com trusty-security/restricted Sources
Hit http://extras.ubuntu.com trusty Release
Hit http://ppa.launchpad.net trusty/main Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/main Sources
Hit http://security.ubuntu.com trusty-security/universe Sources
Hit http://es.archive.ubuntu.com trusty-updates/restricted Sources
Hit http://extras.ubuntu.com trusty/main Sources
Hit http://security.ubuntu.com trusty-security/multiverse Sources
Hit https://archive.cloudera.com trusty-cm5.12.0 InRelease
Hit http://es.archive.ubuntu.com trusty-updates/universe Sources
Hit https://archive.cloudera.com trusty-cm5.12.0/contrib amd64 Packages
Hit http://extras.ubuntu.com trusty/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en_US
Get:1 https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US
Hit http://es.archive.ubuntu.com trusty-updates/multiverse Sources
Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en
Hit http://security.ubuntu.com trusty-security/restricted amd64 Packages
Hit http://extras.ubuntu.com trusty/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/restricted amd64 Packages
Hit http://security.ubuntu.com trusty-security/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/universe amd64 Packages
Hit http://security.ubuntu.com trusty-security/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/main i386 Packages
Hit http://security.ubuntu.com trusty-security/restricted i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/restricted i386 Packages
Hit http://security.ubuntu.com trusty-security/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/universe i386 Packages
Hit http://security.ubuntu.com trusty-security/multiverse i386 Packages
Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US
Hit http://es.archive.ubuntu.com trusty-updates/multiverse i386 Packages
Hit http://security.ubuntu.com trusty-security/main Translation-en
Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://security.ubuntu.com trusty-security/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/multiverse Translation-en
Hit http://security.ubuntu.com trusty-security/restricted Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/restricted Translation-en
Ign http://extras.ubuntu.com trusty/main Translation-en_US
Hit http://es.archive.ubuntu.com trusty-updates/universe Translation-en
Ign http://extras.ubuntu.com trusty/main Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/main Sources
Hit http://es.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://es.archive.ubuntu.com trusty-backports/universe Sources
Hit http://es.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://es.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/restricted i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/multiverse i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/main Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/restricted Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/universe Translation-en
Hit http://es.archive.ubuntu.com trusty Release
Hit http://es.archive.ubuntu.com trusty/main Sources
Hit http://es.archive.ubuntu.com trusty/restricted Sources
Hit http://es.archive.ubuntu.com trusty/universe Sources
Hit http://es.archive.ubuntu.com trusty/multiverse Sources
Hit http://es.archive.ubuntu.com trusty/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://es.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://es.archive.ubuntu.com trusty/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty/main i386 Packages
Hit http://es.archive.ubuntu.com trusty/restricted i386 Packages
Hit http://es.archive.ubuntu.com trusty/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty/multiverse i386 Packages
Hit http://es.archive.ubuntu.com trusty/main Translation-en
Hit http://es.archive.ubuntu.com trusty/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty/restricted Translation-en
Hit http://es.archive.ubuntu.com trusty/universe Translation-en
Ign http://es.archive.ubuntu.com trusty/main Translation-en_US
Ign http://es.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://es.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://es.archive.ubuntu.com trusty/universe Translation-en_US
Reading package lists...
END (0)
BEGIN sudo apt-get update
Hit http://archive.cloudera.com trusty-cm5 InRelease
Hit http://archive.cloudera.com trusty-cm5/contrib Sources
Hit http://ppa.launchpad.net trusty InRelease
Ign http://es.archive.ubuntu.com trusty InRelease
Ign http://extras.ubuntu.com trusty InRelease
Hit http://archive.cloudera.com trusty-cm5/contrib amd64 Packages
Hit http://ppa.launchpad.net trusty/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates InRelease
Hit http://extras.ubuntu.com trusty Release.gpg
Hit http://ppa.launchpad.net trusty/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports InRelease
Hit http://extras.ubuntu.com trusty Release
Hit http://ppa.launchpad.net trusty/main Translation-en
Hit http://es.archive.ubuntu.com trusty Release.gpg
Hit https://archive.cloudera.com trusty-cm5.12.0 InRelease
Hit http://extras.ubuntu.com trusty/main Sources
Hit http://es.archive.ubuntu.com trusty-updates/main Sources
Hit https://archive.cloudera.com trusty-cm5.12.0/contrib amd64 Packages
Hit http://security.ubuntu.com trusty-security InRelease
Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en_US
Get:1 https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US
Hit http://extras.ubuntu.com trusty/main amd64 Packages
Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/restricted Sources
Hit http://extras.ubuntu.com trusty/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/universe Sources
Hit http://es.archive.ubuntu.com trusty-updates/multiverse Sources
Hit http://security.ubuntu.com trusty-security/main Sources
Hit http://es.archive.ubuntu.com trusty-updates/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/restricted amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/universe amd64 Packages
Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US
Hit http://security.ubuntu.com trusty-security/restricted Sources
Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/restricted i386 Packages
Hit http://security.ubuntu.com trusty-security/universe Sources
Hit http://es.archive.ubuntu.com trusty-updates/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty-updates/multiverse i386 Packages
Ign http://extras.ubuntu.com trusty/main Translation-en_US
Hit http://es.archive.ubuntu.com trusty-updates/main Translation-en
Ign http://extras.ubuntu.com trusty/main Translation-en
Hit http://security.ubuntu.com trusty-security/multiverse Sources
Hit http://es.archive.ubuntu.com trusty-updates/multiverse Translation-en
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty-updates/restricted Translation-en
Hit http://es.archive.ubuntu.com trusty-updates/universe Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/main Sources
Hit http://security.ubuntu.com trusty-security/restricted amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://es.archive.ubuntu.com trusty-backports/universe Sources
Hit http://es.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://security.ubuntu.com trusty-security/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty-backports/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/restricted i386 Packages
Hit http://security.ubuntu.com trusty-security/main i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/multiverse i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/main Translation-en
Hit http://security.ubuntu.com trusty-security/restricted i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty-backports/restricted Translation-en
Hit http://security.ubuntu.com trusty-security/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty-backports/universe Translation-en
Hit http://es.archive.ubuntu.com trusty Release
Hit http://security.ubuntu.com trusty-security/multiverse i386 Packages
Hit http://es.archive.ubuntu.com trusty/main Sources
Hit http://es.archive.ubuntu.com trusty/restricted Sources
Hit http://es.archive.ubuntu.com trusty/universe Sources
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://es.archive.ubuntu.com trusty/multiverse Sources
Hit http://es.archive.ubuntu.com trusty/main amd64 Packages
Hit http://es.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://security.ubuntu.com trusty-security/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://es.archive.ubuntu.com trusty/multiverse amd64 Packages
Hit http://es.archive.ubuntu.com trusty/main i386 Packages
Hit http://security.ubuntu.com trusty-security/restricted Translation-en
Hit http://es.archive.ubuntu.com trusty/restricted i386 Packages
Hit http://es.archive.ubuntu.com trusty/universe i386 Packages
Hit http://es.archive.ubuntu.com trusty/multiverse i386 Packages
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Hit http://es.archive.ubuntu.com trusty/main Translation-en
Hit http://es.archive.ubuntu.com trusty/multiverse Translation-en
Hit http://es.archive.ubuntu.com trusty/restricted Translation-en
Hit http://es.archive.ubuntu.com trusty/universe Translation-en
Ign http://es.archive.ubuntu.com trusty/main Translation-en_US
Ign http://es.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://es.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://es.archive.ubuntu.com trusty/universe Translation-en_US
Reading package lists...
END (0)
Installing cloudera-manager-agent package...
BEGIN sudo dpkg -l cloudera-manager-agent | grep -E '^ii[[:space:]]*cloudera-manager-agent[[:space:]]*'
ii cloudera-manager-agent 5.12.0-1.cm5120.p0.120~trusty-cm5 amd64 The Cloudera Manager Agent
END (0)
BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-agent
END (0)
oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons
BEGIN sudo apt-cache show cloudera-manager-agent
Package: cloudera-manager-agent
Source: enterprise
Version: 5.12.0-1.cm5120.p0.120~trusty-cm5
Architecture: amd64
Maintainer: Cloudera Inc. <https://issues.cloudera.org>
Installed-Size: 41468
Depends: libc6 (>= 2.17), libcomerr2 (>= 1.01), libgcc1 (>= 1:4.1.1), libgssapi-krb5-2 (>= 1.10+dfsg~), libkrb5-3 (>= 1.6.dfsg.2), libssl1.0.0 (>= 1.0.1), libstdc++6 (>= 4.6), zlib1g (>= 1:1.2.0), lsb-base, psmisc, bash, libsasl2-modules, libsasl2-modules-gssapi-mit, libxslt1.1, libsqlite3-0, openssl, libssl-dev, libfuse2, fuse-utils | fuse, rpcbind, cloudera-manager-daemons (= 5.12.0-1.cm5120.p0.120~trusty-cm5), perl, python-psycopg2, python-mysqldb, apache2, iproute2
Homepage: http://www.cloudera.com
Priority: extra
Section: misc
Filename: pool/contrib/e/enterprise/cloudera-manager-agent_5.12.0-1.cm5120.p0.120~trusty-cm5_amd64.deb
Size: 9757280
SHA256: c5204a5ae2e3a21cb1c34e7b4472d5a4e7f074d53cdeab9d1a2a28b2504f8d7d
SHA1: cf7d444edcaef6144c63c3c7376414942c047c0e
MD5sum: b47a11f7c8221f4e1819e51c44e347fe
Description: The Cloudera Manager Agent
The Agent is deployed to machines running services managed by Cloudera Manager.
Description-md5: 482a592ee859d31443f58aa3e48fe303

END (0)
Version: 5.12.0-1.cm5120.p0.120~trusty-cm5
BEGIN sudo apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-agent
Reading package lists...
Building dependency tree...
Reading state information...
cloudera-manager-agent is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 584 not upgraded.
END (0)
remote package cloudera-manager-agent installed
Installing cloudera-manager-daemons package...
BEGIN sudo dpkg -l cloudera-manager-daemons | grep -E '^ii[[:space:]]*cloudera-manager-daemons[[:space:]]*'
ii cloudera-manager-daemons 5.12.0-1.cm5120.p0.120~trusty-cm5 all Provides daemons for monitoring Hadoop and related tools.
END (0)
BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-daemons
oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons
END (0)
BEGIN sudo apt-cache show cloudera-manager-daemons
Package: cloudera-manager-daemons
Source: enterprise
Version: 5.12.0-1.cm5120.p0.120~trusty-cm5
Architecture: all
Maintainer: Cloudera Inc. <https://issues.cloudera.org>
Installed-Size: 870556
Depends: perl
Homepage: http://www.cloudera.com
Priority: extra
Section: misc
Filename: pool/contrib/e/enterprise/cloudera-manager-daemons_5.12.0-1.cm5120.p0.120~trusty-cm5_all.deb
Size: 743034520
SHA256: 2098af13c70588b765e759b613948a52673fb3cc8ac688d912b8315099361ce4
SHA1: d59c7c1d7df42592707011922a2c27554b03f7eb
MD5sum: ec945225a977da256e85a3ae28e54766
Description: Provides daemons for monitoring Hadoop and related tools.
Provides daemons for monitoring Hadoop and related tools.
Description-md5: 35bb5bc8fc2f296f791c3bda9e345340

END (0)
Version: 5.12.0-1.cm5120.p0.120~trusty-cm5
BEGIN sudo apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-daemons
Reading package lists...
Building dependency tree...
Reading state information...
cloudera-manager-daemons is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 584 not upgraded.
END (0)
remote package cloudera-manager-daemons installed
Installing Unlimited Strength Encryption policy files.
Installation not requested. Step will be skipped.
Configuring Cloudera Manager Agent...
BEGIN grep server_host=192.168.XXX.XXX /etc/cloudera-scm-agent/config.ini
server_host=192.168.XXX.XXX
END (0)
scm agent is already configured
Starting Cloudera Manager Agent...
BEGIN /usr/sbin/service cloudera-scm-agent status
Checking for service cloudera-scm-agent * cloudera-scm-agent is not running
END (3)
BEGIN sudo /usr/sbin/service cloudera-scm-agent start
Starting cloudera-scm-agent: bash: /var/log/cloudera-scm-agent/cloudera-scm-agent.out: Permission denied
* Couldn't start cloudera-scm-agent

END (0)
agent logs:
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/'
>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration
>> global REDACTOR
>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration
>> global NOP_REDACTOR
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration
>> global REDACTOR
>>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration
>> global NOP_REDACTOR
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO SCM Agent Version: 5.12.0
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755
>>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent
END (0)
BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/'
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Found cgroups subsystem: cpuacct
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Found cgroups subsystem: memory
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Found cgroups subsystem: blkio
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/memory
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpu
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpuacct
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/blkio
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': -1, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True}
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Setting up supervisord event monitor.
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Using timeout of 2.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Using join timeout of 0.100000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Using tolerance of 60.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4', 'xfs']
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread kt_renewer INFO Agent wide credential cache set to /run/cloudera-scm-agent/krb5cc_cm_agent_0
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Using max_collection_wait_seconds of 10.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/monitor/tasktracker/schema.json
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread ntp_monitor INFO Using timeout of 2.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread dns_names INFO Using timeout of 30.000000
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread __init__ INFO Created DNS monitor.
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread __init__ INFO Importing metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/monitor/schema.json
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'ORACLE_HOME': '/usr/share/oracle/instantclient', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/process
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/flood
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent        ERROR    Failed to connect to previous supervisor.
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2109, in find_or_start_supervisor
>> self.configure_supervisor_clients()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2290, in configure_supervisor_clients
>> supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")])
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 1599, in realize
>> Options.realize(self, *arg, **kw)
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 333, in realize
>> self.process_config()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 341, in process_config
>> self.process_config_file(do_usage)
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 376, in process_config_file
>> self.usage(str(msg))
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 164, in usage
>> self.exit(2)
>>SystemExit: 2
>>[07/Aug/2017 10:25:17 +0000] 10429 Dummy-1 daemonize WARNING Stopping daemon.
>>[07/Aug/2017 10:25:17 +0000] 10429 Dummy-1 agent INFO Stopping agent...
>>[07/Aug/2017 10:25:17 +0000] 10429 Dummy-1 agent INFO No extant cgroups; unmounting any cgroup roots
END (0)
end of agent logs.
scm agent started 

 

I tried reinstalling and so on, but I can't solve this. This is second time I try to install Cloudera Manager. First one was in a single node and it went all right at first time.


Anyone have any idea on how to solve this?


Thanks you so much in advance.

 

1 ACCEPTED SOLUTION

avatar
Master Collaborator

If running as sudo the results is giving you 

 

OSError: [Errno 13] Permission denied: '/run/user/112/gvfs'

You will need to temporary unmount that tmpfs mount/path, which will allow the Cloudera Manager Agent to start - you are hitting a known issue which will be addressed in the future maintenance release.

 

Regards,

 

Michalis

View solution in original post

14 REPLIES 14

avatar
Master Collaborator

Hello Josholsan,

 

This stack trace [0] suggests that Cloudera Manager Agent failed to create '/var/run/cloudera-scm-agent/supervisor/supervisord.conf', from past experience the error is often misleading. 

 

To assist us with troubleshooting;

 

On the affected node can you provide the output of the following command;

 

[bash prompt root-user]# /usr/lib/cmf/agent/build/env/bin/python /usr/lib/cmf/agent/build/env/bin/cmf-agent --package_dir /usr/lib/cmf/service --agent_dir /var/run/cloudera-scm-agent --lib_dir /var/lib/cloudera-scm-agent --logfile /var/log/cloudera-scm-agent/cloudera-scm-agent.log --comm_name cmf-agent --pidfile /var/run/cloudera-scm-agent/cloudera-scm-agent.pid

 

Best,

 

Michalis

 

[0] 

>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent        ERROR    Failed to connect to previous supervisor.
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2109, in find_or_start_supervisor
>> self.configure_supervisor_clients()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2290, in configure_supervisor_clients
>> supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")])
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 1599, in realize
>> Options.realize(self, *arg, **kw)
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 333, in realize
>> self.process_config()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 341, in process_config
>> self.process_config_file(do_usage)
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 376, in process_config_file
>> self.usage(str(msg))
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 164, in usage
>> self.exit(2)
>>SystemExit: 2

 

 

avatar
Explorer

Good morning Michalis,

 

this is happening in all my three nodes, they are stuck in the same point. The result for the command you said to me is the following one in al three nodes:

 

[09/Aug/2017 08:22:09 +0000] 2459 MainThread agent        INFO     SCM Agent Version: 5.12.0
[09/Aug/2017 08:22:09 +0000] 2459 MainThread agent        WARNING  Expected mode 0751 for /run/cloudera-scm-agent but was 0755
[09/Aug/2017 08:22:09 +0000] 2459 MainThread agent        INFO     Re-using pre-existing directory: /run/cloudera-scm-agent
[09/Aug/2017 08:22:09 +0000] 2459 MainThread agent        INFO     Failed to write pid file: /var/run/cloudera-scm-agent/cloudera-scm-agent.pid
Failed to write pid file: /var/run/cloudera-scm-agent/cloudera-scm-agent.pid

 

I also tried to execute this command as sudo, the result is :

 

[09/Aug/2017 10:20:09 +0000] 8013 MainThread agent        INFO     SCM Agent Version: 5.12.0
[09/Aug/2017 10:20:09 +0000] 8013 MainThread agent        WARNING  Expected mode 0751 for /run/cloudera-scm-agent but was 0755
[09/Aug/2017 10:20:09 +0000] 8013 MainThread agent        INFO     Re-using pre-existing directory: /run/cloudera-scm-agent
[09/Aug/2017 10:20:09 +0000] 8013 MainThread agent        INFO     Adding env vars that start with CMF_AGENT_
[09/Aug/2017 10:20:09 +0000] 8013 MainThread agent        INFO     Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
Error: could not find config file /run/cloudera-scm-agent/supervisor/supervisord.conf
For help, use /usr/lib/cmf/agent/build/env/bin/cmf-agent -h
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/bin/cmf-agent", line 12, in <module>
    load_entry_point('cmf==5.12.0', 'console_scripts', 'cmf-agent')()
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 3095, in main
    main_impl()
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 3078, in main_impl
    agent.start()
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 804, in start
    self.find_or_start_supervisor()
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2151, in find_or_start_supervisor
    if not mount_tmpfs(process_dir, self.args.clear_agent_dir, self.sudo):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/tmpfs.py", line 62, in mount_tmpfs
    if os.path.samefile(p.mountpoint, path) and p.fstype == "tmpfs" and "noexec" not in p.opts:
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/posixpath.py", line 162, in samefile
    s1 = os.stat(f1)
OSError: [Errno 13] Permission denied: '/run/user/112/gvfs'

 

Thanks you so much,

Jose.

avatar
Master Collaborator

If running as sudo the results is giving you 

 

OSError: [Errno 13] Permission denied: '/run/user/112/gvfs'

You will need to temporary unmount that tmpfs mount/path, which will allow the Cloudera Manager Agent to start - you are hitting a known issue which will be addressed in the future maintenance release.

 

Regards,

 

Michalis

avatar
Explorer

Thanks you so much Michalis. Unmount that tmpfs solved the problem for me. This tmpfs is used for something? I mean, shall I mount this after installation?

 

Thanks you so much again! You made my day!!

avatar
Master Collaborator

The mountpoint isn't part of Cloudera, and likely being mounted by other process in your system. The process that controls the tmpfs mount should be able to remount it again, however you will run into the issue again if you hard_stop the agent. The fix is already been committed and will be available in the next Cloudera maintenance release.

avatar
Explorer
hi! i have the same problem with,could you give some details about how to resolve it? thks.

avatar
Explorer

Does not the Michalis' provided solution worked for you?

 

In this case:

 

1. Can we see the content of your /etc/hosts file?

2. Can we see the conten of your /etc/hostname file?

3. What are the result of the following commands execution?:

 

-$ hostname

-$ hostname -f

-$ python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'

 

Any log errors?

 

Thanks.

avatar
Explorer

[root@bigdata-master cloudera-scm-agent]# hostname
bigdata-master
[root@bigdata-master cloudera-scm-agent]# hostname -f
bigdata-master.lcloud.com
[root@bigdata-master cloudera-scm-agent]# python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'
bigdata-master.lcloud.com 192.168.1.127
[root@bigdata-master cloudera-scm-agent]#

 

====

ths,i run the command and got this,is this any problem?

avatar
Champion

could you share your /etc/hosts ? 

status of your firewall ? 

what os are you runining  ?