Created on 08-08-2017 03:28 AM - edited 08-08-2017 05:31 AM
Hello all,
I've been searching in the forum for this error. I know this is a very common error and I've read a lot of differents posts with related information and solutions. But I still can't solve the problem. I've been trying for two days, and maybe you can help me with something I'm passing through.
The point is that I'm trying to install a 3 nodes cluster with Cloudera Manager in Ubuntu 14.04 LTS using the Installation Path A. In the Cluster installation I receive the following error:
Installation failed. Failed to receive heartbeat from agent. Ensure that the host's hostname is configured properly. Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules). Ensure that ports 9000 and 9001 are not in use on the host being added. Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details). If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.
All ports are open. Firewall and SeLinux are deactivated and the following commands with their results are:
hostname
vbigdata0srv.domain.com
hostname -f
vbigdata0srv.domain.com
python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'
vbigdata0srv.domain.com 192.168.XXX.XXX (The right IP)
*This is the same for all 3 nodes (vbigdata1srv and vbigdata2srv) with their respective IPs.
The log looks like this, showing as error: ">>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent ERROR Failed to connect to previous supervisor. "
/tmp/scm_prepare_node.25wJDbEq using SSH_CLIENT to get the SCM hostname: 192.168.XXX.XXX 38281 22 opening logging file descriptor Starting installation script... Acquiring installation lock... BEGIN flock 4 END (0) Detecting root privileges... effective UID is 1200 BEGIN which pbrun END (1) BEGIN sudo -S id uid=0(root) gid=0(root) groups=0(root) END (0) Using 'sudo ' to acquire root privileges Detecting distribution... BEGIN grep 'Ubuntu' /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_DESCRIPTION="Ubuntu 14.04 LTS" END (0) BEGIN grep DISTRIB_CODENAME /etc/lsb-release DISTRIB_CODENAME=trusty END (0) BEGIN echo DISTRIB_CODENAME=trusty | cut -d = -f 2 trusty END (0) Detecting Cloudera Manager Server... BEGIN host -t PTR 192.168.XXX.XXX Host XXX.XXX.168.192.in-addr.arpa. not found: 3(NXDOMAIN) END (1) BEGIN which python END (0) BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' 192.168.XXX.XXX 7182 /usr/bin/python END (0) BEGIN which wget END (0) /usr/bin/wget BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo END (4) Installing package repositories... validating format of repository file /tmp/scm_prepare_node.25wJDbEq/repos/ubuntu_trusty/cloudera-manager.list installing repository file /tmp/scm_prepare_node.25wJDbEq/repos/ubuntu_trusty/cloudera-manager.list repository file /tmp/scm_prepare_node.25wJDbEq/repos/ubuntu_trusty/cloudera-manager.list installed BEGIN sudo apt-key add /tmp/scm_prepare_node.25wJDbEq/archive.key OK END (0) installing priority file /tmp/scm_prepare_node.25wJDbEq/priorities/ubuntu_trusty/cloudera.pref priority file /tmp/scm_prepare_node.25wJDbEq/priorities/ubuntu_trusty/cloudera.pref installed Refreshing package metadata... BEGIN sudo apt-get update Hit http://ppa.launchpad.net trusty InRelease Ign http://es.archive.ubuntu.com trusty InRelease Hit http://security.ubuntu.com trusty-security InRelease Hit http://archive.cloudera.com trusty-cm5 InRelease Ign http://extras.ubuntu.com trusty InRelease Hit http://es.archive.ubuntu.com trusty-updates InRelease Hit http://ppa.launchpad.net trusty/main amd64 Packages Hit http://archive.cloudera.com trusty-cm5/contrib Sources Hit http://es.archive.ubuntu.com trusty-backports InRelease Hit http://extras.ubuntu.com trusty Release.gpg Hit http://security.ubuntu.com trusty-security/main Sources Hit http://archive.cloudera.com trusty-cm5/contrib amd64 Packages Hit http://ppa.launchpad.net trusty/main i386 Packages Hit http://es.archive.ubuntu.com trusty Release.gpg Hit http://security.ubuntu.com trusty-security/restricted Sources Hit http://extras.ubuntu.com trusty Release Hit http://ppa.launchpad.net trusty/main Translation-en Hit http://es.archive.ubuntu.com trusty-updates/main Sources Hit http://security.ubuntu.com trusty-security/universe Sources Hit http://es.archive.ubuntu.com trusty-updates/restricted Sources Hit http://extras.ubuntu.com trusty/main Sources Hit http://security.ubuntu.com trusty-security/multiverse Sources Hit https://archive.cloudera.com trusty-cm5.12.0 InRelease Hit http://es.archive.ubuntu.com trusty-updates/universe Sources Hit https://archive.cloudera.com trusty-cm5.12.0/contrib amd64 Packages Hit http://extras.ubuntu.com trusty/main amd64 Packages Hit http://security.ubuntu.com trusty-security/main amd64 Packages Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en_US Get:1 https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US Hit http://es.archive.ubuntu.com trusty-updates/multiverse Sources Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en Hit http://security.ubuntu.com trusty-security/restricted amd64 Packages Hit http://extras.ubuntu.com trusty/main i386 Packages Hit http://es.archive.ubuntu.com trusty-updates/main amd64 Packages Hit http://security.ubuntu.com trusty-security/universe amd64 Packages Hit http://es.archive.ubuntu.com trusty-updates/restricted amd64 Packages Hit http://security.ubuntu.com trusty-security/multiverse amd64 Packages Hit http://es.archive.ubuntu.com trusty-updates/universe amd64 Packages Hit http://security.ubuntu.com trusty-security/main i386 Packages Hit http://es.archive.ubuntu.com trusty-updates/multiverse amd64 Packages Hit http://es.archive.ubuntu.com trusty-updates/main i386 Packages Hit http://security.ubuntu.com trusty-security/restricted i386 Packages Hit http://es.archive.ubuntu.com trusty-updates/restricted i386 Packages Hit http://security.ubuntu.com trusty-security/universe i386 Packages Hit http://es.archive.ubuntu.com trusty-updates/universe i386 Packages Hit http://security.ubuntu.com trusty-security/multiverse i386 Packages Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US Hit http://es.archive.ubuntu.com trusty-updates/multiverse i386 Packages Hit http://security.ubuntu.com trusty-security/main Translation-en Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en Hit http://es.archive.ubuntu.com trusty-updates/main Translation-en Hit http://security.ubuntu.com trusty-security/multiverse Translation-en Hit http://es.archive.ubuntu.com trusty-updates/multiverse Translation-en Hit http://security.ubuntu.com trusty-security/restricted Translation-en Hit http://security.ubuntu.com trusty-security/universe Translation-en Hit http://es.archive.ubuntu.com trusty-updates/restricted Translation-en Ign http://extras.ubuntu.com trusty/main Translation-en_US Hit http://es.archive.ubuntu.com trusty-updates/universe Translation-en Ign http://extras.ubuntu.com trusty/main Translation-en Hit http://es.archive.ubuntu.com trusty-backports/main Sources Hit http://es.archive.ubuntu.com trusty-backports/restricted Sources Hit http://es.archive.ubuntu.com trusty-backports/universe Sources Hit http://es.archive.ubuntu.com trusty-backports/multiverse Sources Hit http://es.archive.ubuntu.com trusty-backports/main amd64 Packages Hit http://es.archive.ubuntu.com trusty-backports/restricted amd64 Packages Hit http://es.archive.ubuntu.com trusty-backports/universe amd64 Packages Hit http://es.archive.ubuntu.com trusty-backports/multiverse amd64 Packages Hit http://es.archive.ubuntu.com trusty-backports/main i386 Packages Hit http://es.archive.ubuntu.com trusty-backports/restricted i386 Packages Hit http://es.archive.ubuntu.com trusty-backports/universe i386 Packages Hit http://es.archive.ubuntu.com trusty-backports/multiverse i386 Packages Hit http://es.archive.ubuntu.com trusty-backports/main Translation-en Hit http://es.archive.ubuntu.com trusty-backports/multiverse Translation-en Hit http://es.archive.ubuntu.com trusty-backports/restricted Translation-en Hit http://es.archive.ubuntu.com trusty-backports/universe Translation-en Hit http://es.archive.ubuntu.com trusty Release Hit http://es.archive.ubuntu.com trusty/main Sources Hit http://es.archive.ubuntu.com trusty/restricted Sources Hit http://es.archive.ubuntu.com trusty/universe Sources Hit http://es.archive.ubuntu.com trusty/multiverse Sources Hit http://es.archive.ubuntu.com trusty/main amd64 Packages Hit http://es.archive.ubuntu.com trusty/restricted amd64 Packages Hit http://es.archive.ubuntu.com trusty/universe amd64 Packages Hit http://es.archive.ubuntu.com trusty/multiverse amd64 Packages Hit http://es.archive.ubuntu.com trusty/main i386 Packages Hit http://es.archive.ubuntu.com trusty/restricted i386 Packages Hit http://es.archive.ubuntu.com trusty/universe i386 Packages Hit http://es.archive.ubuntu.com trusty/multiverse i386 Packages Hit http://es.archive.ubuntu.com trusty/main Translation-en Hit http://es.archive.ubuntu.com trusty/multiverse Translation-en Hit http://es.archive.ubuntu.com trusty/restricted Translation-en Hit http://es.archive.ubuntu.com trusty/universe Translation-en Ign http://es.archive.ubuntu.com trusty/main Translation-en_US Ign http://es.archive.ubuntu.com trusty/multiverse Translation-en_US Ign http://es.archive.ubuntu.com trusty/restricted Translation-en_US Ign http://es.archive.ubuntu.com trusty/universe Translation-en_US Reading package lists... END (0) BEGIN sudo apt-get update Hit http://archive.cloudera.com trusty-cm5 InRelease Hit http://archive.cloudera.com trusty-cm5/contrib Sources Hit http://ppa.launchpad.net trusty InRelease Ign http://es.archive.ubuntu.com trusty InRelease Ign http://extras.ubuntu.com trusty InRelease Hit http://archive.cloudera.com trusty-cm5/contrib amd64 Packages Hit http://ppa.launchpad.net trusty/main amd64 Packages Hit http://es.archive.ubuntu.com trusty-updates InRelease Hit http://extras.ubuntu.com trusty Release.gpg Hit http://ppa.launchpad.net trusty/main i386 Packages Hit http://es.archive.ubuntu.com trusty-backports InRelease Hit http://extras.ubuntu.com trusty Release Hit http://ppa.launchpad.net trusty/main Translation-en Hit http://es.archive.ubuntu.com trusty Release.gpg Hit https://archive.cloudera.com trusty-cm5.12.0 InRelease Hit http://extras.ubuntu.com trusty/main Sources Hit http://es.archive.ubuntu.com trusty-updates/main Sources Hit https://archive.cloudera.com trusty-cm5.12.0/contrib amd64 Packages Hit http://security.ubuntu.com trusty-security InRelease Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en_US Get:1 https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US Hit http://extras.ubuntu.com trusty/main amd64 Packages Ign http://archive.cloudera.com trusty-cm5/contrib Translation-en Hit http://es.archive.ubuntu.com trusty-updates/restricted Sources Hit http://extras.ubuntu.com trusty/main i386 Packages Hit http://es.archive.ubuntu.com trusty-updates/universe Sources Hit http://es.archive.ubuntu.com trusty-updates/multiverse Sources Hit http://security.ubuntu.com trusty-security/main Sources Hit http://es.archive.ubuntu.com trusty-updates/main amd64 Packages Hit http://es.archive.ubuntu.com trusty-updates/restricted amd64 Packages Hit http://es.archive.ubuntu.com trusty-updates/universe amd64 Packages Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en_US Hit http://security.ubuntu.com trusty-security/restricted Sources Ign https://archive.cloudera.com trusty-cm5.12.0/contrib Translation-en Hit http://es.archive.ubuntu.com trusty-updates/multiverse amd64 Packages Hit http://es.archive.ubuntu.com trusty-updates/main i386 Packages Hit http://es.archive.ubuntu.com trusty-updates/restricted i386 Packages Hit http://security.ubuntu.com trusty-security/universe Sources Hit http://es.archive.ubuntu.com trusty-updates/universe i386 Packages Hit http://es.archive.ubuntu.com trusty-updates/multiverse i386 Packages Ign http://extras.ubuntu.com trusty/main Translation-en_US Hit http://es.archive.ubuntu.com trusty-updates/main Translation-en Ign http://extras.ubuntu.com trusty/main Translation-en Hit http://security.ubuntu.com trusty-security/multiverse Sources Hit http://es.archive.ubuntu.com trusty-updates/multiverse Translation-en Hit http://security.ubuntu.com trusty-security/main amd64 Packages Hit http://es.archive.ubuntu.com trusty-updates/restricted Translation-en Hit http://es.archive.ubuntu.com trusty-updates/universe Translation-en Hit http://es.archive.ubuntu.com trusty-backports/main Sources Hit http://security.ubuntu.com trusty-security/restricted amd64 Packages Hit http://es.archive.ubuntu.com trusty-backports/restricted Sources Hit http://es.archive.ubuntu.com trusty-backports/universe Sources Hit http://es.archive.ubuntu.com trusty-backports/multiverse Sources Hit http://security.ubuntu.com trusty-security/universe amd64 Packages Hit http://es.archive.ubuntu.com trusty-backports/main amd64 Packages Hit http://es.archive.ubuntu.com trusty-backports/restricted amd64 Packages Hit http://es.archive.ubuntu.com trusty-backports/universe amd64 Packages Hit http://security.ubuntu.com trusty-security/multiverse amd64 Packages Hit http://es.archive.ubuntu.com trusty-backports/multiverse amd64 Packages Hit http://es.archive.ubuntu.com trusty-backports/main i386 Packages Hit http://es.archive.ubuntu.com trusty-backports/restricted i386 Packages Hit http://security.ubuntu.com trusty-security/main i386 Packages Hit http://es.archive.ubuntu.com trusty-backports/universe i386 Packages Hit http://es.archive.ubuntu.com trusty-backports/multiverse i386 Packages Hit http://es.archive.ubuntu.com trusty-backports/main Translation-en Hit http://security.ubuntu.com trusty-security/restricted i386 Packages Hit http://es.archive.ubuntu.com trusty-backports/multiverse Translation-en Hit http://es.archive.ubuntu.com trusty-backports/restricted Translation-en Hit http://security.ubuntu.com trusty-security/universe i386 Packages Hit http://es.archive.ubuntu.com trusty-backports/universe Translation-en Hit http://es.archive.ubuntu.com trusty Release Hit http://security.ubuntu.com trusty-security/multiverse i386 Packages Hit http://es.archive.ubuntu.com trusty/main Sources Hit http://es.archive.ubuntu.com trusty/restricted Sources Hit http://es.archive.ubuntu.com trusty/universe Sources Hit http://security.ubuntu.com trusty-security/main Translation-en Hit http://es.archive.ubuntu.com trusty/multiverse Sources Hit http://es.archive.ubuntu.com trusty/main amd64 Packages Hit http://es.archive.ubuntu.com trusty/restricted amd64 Packages Hit http://security.ubuntu.com trusty-security/multiverse Translation-en Hit http://es.archive.ubuntu.com trusty/universe amd64 Packages Hit http://es.archive.ubuntu.com trusty/multiverse amd64 Packages Hit http://es.archive.ubuntu.com trusty/main i386 Packages Hit http://security.ubuntu.com trusty-security/restricted Translation-en Hit http://es.archive.ubuntu.com trusty/restricted i386 Packages Hit http://es.archive.ubuntu.com trusty/universe i386 Packages Hit http://es.archive.ubuntu.com trusty/multiverse i386 Packages Hit http://security.ubuntu.com trusty-security/universe Translation-en Hit http://es.archive.ubuntu.com trusty/main Translation-en Hit http://es.archive.ubuntu.com trusty/multiverse Translation-en Hit http://es.archive.ubuntu.com trusty/restricted Translation-en Hit http://es.archive.ubuntu.com trusty/universe Translation-en Ign http://es.archive.ubuntu.com trusty/main Translation-en_US Ign http://es.archive.ubuntu.com trusty/multiverse Translation-en_US Ign http://es.archive.ubuntu.com trusty/restricted Translation-en_US Ign http://es.archive.ubuntu.com trusty/universe Translation-en_US Reading package lists... END (0) Installing cloudera-manager-agent package... BEGIN sudo dpkg -l cloudera-manager-agent | grep -E '^ii[[:space:]]*cloudera-manager-agent[[:space:]]*' ii cloudera-manager-agent 5.12.0-1.cm5120.p0.120~trusty-cm5 amd64 The Cloudera Manager Agent END (0) BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-agent END (0) oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons BEGIN sudo apt-cache show cloudera-manager-agent Package: cloudera-manager-agent Source: enterprise Version: 5.12.0-1.cm5120.p0.120~trusty-cm5 Architecture: amd64 Maintainer: Cloudera Inc. <https://issues.cloudera.org> Installed-Size: 41468 Depends: libc6 (>= 2.17), libcomerr2 (>= 1.01), libgcc1 (>= 1:4.1.1), libgssapi-krb5-2 (>= 1.10+dfsg~), libkrb5-3 (>= 1.6.dfsg.2), libssl1.0.0 (>= 1.0.1), libstdc++6 (>= 4.6), zlib1g (>= 1:1.2.0), lsb-base, psmisc, bash, libsasl2-modules, libsasl2-modules-gssapi-mit, libxslt1.1, libsqlite3-0, openssl, libssl-dev, libfuse2, fuse-utils | fuse, rpcbind, cloudera-manager-daemons (= 5.12.0-1.cm5120.p0.120~trusty-cm5), perl, python-psycopg2, python-mysqldb, apache2, iproute2 Homepage: http://www.cloudera.com Priority: extra Section: misc Filename: pool/contrib/e/enterprise/cloudera-manager-agent_5.12.0-1.cm5120.p0.120~trusty-cm5_amd64.deb Size: 9757280 SHA256: c5204a5ae2e3a21cb1c34e7b4472d5a4e7f074d53cdeab9d1a2a28b2504f8d7d SHA1: cf7d444edcaef6144c63c3c7376414942c047c0e MD5sum: b47a11f7c8221f4e1819e51c44e347fe Description: The Cloudera Manager Agent The Agent is deployed to machines running services managed by Cloudera Manager. Description-md5: 482a592ee859d31443f58aa3e48fe303 END (0) Version: 5.12.0-1.cm5120.p0.120~trusty-cm5 BEGIN sudo apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-agent Reading package lists... Building dependency tree... Reading state information... cloudera-manager-agent is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 584 not upgraded. END (0) remote package cloudera-manager-agent installed Installing cloudera-manager-daemons package... BEGIN sudo dpkg -l cloudera-manager-daemons | grep -E '^ii[[:space:]]*cloudera-manager-daemons[[:space:]]*' ii cloudera-manager-daemons 5.12.0-1.cm5120.p0.120~trusty-cm5 all Provides daemons for monitoring Hadoop and related tools. END (0) BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-daemons oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons END (0) BEGIN sudo apt-cache show cloudera-manager-daemons Package: cloudera-manager-daemons Source: enterprise Version: 5.12.0-1.cm5120.p0.120~trusty-cm5 Architecture: all Maintainer: Cloudera Inc. <https://issues.cloudera.org> Installed-Size: 870556 Depends: perl Homepage: http://www.cloudera.com Priority: extra Section: misc Filename: pool/contrib/e/enterprise/cloudera-manager-daemons_5.12.0-1.cm5120.p0.120~trusty-cm5_all.deb Size: 743034520 SHA256: 2098af13c70588b765e759b613948a52673fb3cc8ac688d912b8315099361ce4 SHA1: d59c7c1d7df42592707011922a2c27554b03f7eb MD5sum: ec945225a977da256e85a3ae28e54766 Description: Provides daemons for monitoring Hadoop and related tools. Provides daemons for monitoring Hadoop and related tools. Description-md5: 35bb5bc8fc2f296f791c3bda9e345340 END (0) Version: 5.12.0-1.cm5120.p0.120~trusty-cm5 BEGIN sudo apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-daemons Reading package lists... Building dependency tree... Reading state information... cloudera-manager-daemons is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 584 not upgraded. END (0) remote package cloudera-manager-daemons installed Installing Unlimited Strength Encryption policy files. Installation not requested. Step will be skipped. Configuring Cloudera Manager Agent... BEGIN grep server_host=192.168.XXX.XXX /etc/cloudera-scm-agent/config.ini server_host=192.168.XXX.XXX END (0) scm agent is already configured Starting Cloudera Manager Agent... BEGIN /usr/sbin/service cloudera-scm-agent status Checking for service cloudera-scm-agent * cloudera-scm-agent is not running END (3) BEGIN sudo /usr/sbin/service cloudera-scm-agent start Starting cloudera-scm-agent: bash: /var/log/cloudera-scm-agent/cloudera-scm-agent.out: Permission denied * Couldn't start cloudera-scm-agent END (0) agent logs: BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/' >>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration >> global REDACTOR >>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration >> global NOP_REDACTOR >>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO SCM Agent Version: 5.12.0 >>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 >>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent >>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO SCM Agent Version: 5.12.0 >>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 >>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent >>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO SCM Agent Version: 5.12.0 >>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 >>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent >>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO SCM Agent Version: 5.12.0 >>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 >>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent >>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO SCM Agent Version: 5.12.0 >>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 >>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent >>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:260: SyntaxWarning: name 'REDACTOR' is assigned to before global declaration >> global REDACTOR >>/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/redaction.py:261: SyntaxWarning: name 'NOP_REDACTOR' is assigned to before global declaration >> global NOP_REDACTOR >>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO SCM Agent Version: 5.12.0 >>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 >>[07/Aug/2017 10:01:37 +0000] 6868 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent >>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO SCM Agent Version: 5.12.0 >>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 >>[07/Aug/2017 10:09:53 +0000] 7660 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent >>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO SCM Agent Version: 5.12.0 >>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 >>[07/Aug/2017 10:12:46 +0000] 8288 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent >>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO SCM Agent Version: 5.12.0 >>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 >>[07/Aug/2017 10:17:36 +0000] 8904 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent >>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO SCM Agent Version: 5.12.0 >>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 >>[07/Aug/2017 10:25:15 +0000] 10415 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent END (0) BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/' >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Found cgroups subsystem: cpuacct >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Found cgroups subsystem: memory >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Found cgroups subsystem: blkio >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/memory >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpu >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpuacct >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/blkio >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': -1, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True} >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Setting up supervisord event monitor. >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Using timeout of 2.000000 >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Using join timeout of 0.100000 >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Using tolerance of 60.000000 >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4', 'xfs'] >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread kt_renewer INFO Agent wide credential cache set to /run/cloudera-scm-agent/krb5cc_cm_agent_0 >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/monitor/tasktracker/schema.json >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread ntp_monitor INFO Using timeout of 2.000000 >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread dns_names INFO Using timeout of 30.000000 >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread __init__ INFO Created DNS monitor. >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880 >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread __init__ INFO Importing metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/monitor/schema.json >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'ORACLE_HOME': '/usr/share/oracle/instantclient', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'} >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/process >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/flood >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include >>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent ERROR Failed to connect to previous supervisor. >>Traceback (most recent call last): >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2109, in find_or_start_supervisor >> self.configure_supervisor_clients() >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2290, in configure_supervisor_clients >> supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")]) >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 1599, in realize >> Options.realize(self, *arg, **kw) >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 333, in realize >> self.process_config() >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 341, in process_config >> self.process_config_file(do_usage) >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 376, in process_config_file >> self.usage(str(msg)) >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 164, in usage >> self.exit(2) >>SystemExit: 2 >>[07/Aug/2017 10:25:17 +0000] 10429 Dummy-1 daemonize WARNING Stopping daemon. >>[07/Aug/2017 10:25:17 +0000] 10429 Dummy-1 agent INFO Stopping agent... >>[07/Aug/2017 10:25:17 +0000] 10429 Dummy-1 agent INFO No extant cgroups; unmounting any cgroup roots END (0) end of agent logs. scm agent started
I tried reinstalling and so on, but I can't solve this. This is second time I try to install Cloudera Manager. First one was in a single node and it went all right at first time.
Anyone have any idea on how to solve this?
Thanks you so much in advance.
Created on 08-09-2017 03:02 AM - edited 08-09-2017 11:51 AM
If running as sudo the results is giving you
OSError: [Errno 13] Permission denied: '/run/user/112/gvfs'
You will need to temporary unmount that tmpfs mount/path, which will allow the Cloudera Manager Agent to start - you are hitting a known issue which will be addressed in the future maintenance release.
Regards,
Michalis
Created 08-08-2017 06:51 AM
Hello Josholsan,
This stack trace [0] suggests that Cloudera Manager Agent failed to create '/var/run/cloudera-scm-agent/supervisor/supervisord.conf', from past experience the error is often misleading.
To assist us with troubleshooting;
On the affected node can you provide the output of the following command;
[bash prompt root-user]# /usr/lib/cmf/agent/build/env/bin/python /usr/lib/cmf/agent/build/env/bin/cmf-agent --package_dir /usr/lib/cmf/service --agent_dir /var/run/cloudera-scm-agent --lib_dir /var/lib/cloudera-scm-agent --logfile /var/log/cloudera-scm-agent/cloudera-scm-agent.log --comm_name cmf-agent --pidfile /var/run/cloudera-scm-agent/cloudera-scm-agent.pid
Best,
Michalis
[0]
>>[07/Aug/2017 10:25:17 +0000] 10429 MainThread agent ERROR Failed to connect to previous supervisor. >>Traceback (most recent call last): >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2109, in find_or_start_supervisor >> self.configure_supervisor_clients() >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2290, in configure_supervisor_clients >> supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")]) >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 1599, in realize >> Options.realize(self, *arg, **kw) >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 333, in realize >> self.process_config() >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 341, in process_config >> self.process_config_file(do_usage) >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 376, in process_config_file >> self.usage(str(msg)) >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/options.py", line 164, in usage >> self.exit(2) >>SystemExit: 2
Created on 08-08-2017 11:24 PM - edited 08-09-2017 01:21 AM
Good morning Michalis,
this is happening in all my three nodes, they are stuck in the same point. The result for the command you said to me is the following one in al three nodes:
[09/Aug/2017 08:22:09 +0000] 2459 MainThread agent INFO SCM Agent Version: 5.12.0 [09/Aug/2017 08:22:09 +0000] 2459 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [09/Aug/2017 08:22:09 +0000] 2459 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent [09/Aug/2017 08:22:09 +0000] 2459 MainThread agent INFO Failed to write pid file: /var/run/cloudera-scm-agent/cloudera-scm-agent.pid Failed to write pid file: /var/run/cloudera-scm-agent/cloudera-scm-agent.pid
I also tried to execute this command as sudo, the result is :
[09/Aug/2017 10:20:09 +0000] 8013 MainThread agent INFO SCM Agent Version: 5.12.0 [09/Aug/2017 10:20:09 +0000] 8013 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [09/Aug/2017 10:20:09 +0000] 8013 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent [09/Aug/2017 10:20:09 +0000] 8013 MainThread agent INFO Adding env vars that start with CMF_AGENT_ [09/Aug/2017 10:20:09 +0000] 8013 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log Error: could not find config file /run/cloudera-scm-agent/supervisor/supervisord.conf For help, use /usr/lib/cmf/agent/build/env/bin/cmf-agent -h Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/bin/cmf-agent", line 12, in <module> load_entry_point('cmf==5.12.0', 'console_scripts', 'cmf-agent')() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 3095, in main main_impl() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 3078, in main_impl agent.start() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 804, in start self.find_or_start_supervisor() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/agent.py", line 2151, in find_or_start_supervisor if not mount_tmpfs(process_dir, self.args.clear_agent_dir, self.sudo): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.0-py2.7.egg/cmf/tmpfs.py", line 62, in mount_tmpfs if os.path.samefile(p.mountpoint, path) and p.fstype == "tmpfs" and "noexec" not in p.opts: File "/usr/lib/cmf/agent/build/env/lib/python2.7/posixpath.py", line 162, in samefile s1 = os.stat(f1) OSError: [Errno 13] Permission denied: '/run/user/112/gvfs'
Thanks you so much,
Jose.
Created on 08-09-2017 03:02 AM - edited 08-09-2017 11:51 AM
If running as sudo the results is giving you
OSError: [Errno 13] Permission denied: '/run/user/112/gvfs'
You will need to temporary unmount that tmpfs mount/path, which will allow the Cloudera Manager Agent to start - you are hitting a known issue which will be addressed in the future maintenance release.
Regards,
Michalis
Created 08-09-2017 04:04 AM
Thanks you so much Michalis. Unmount that tmpfs solved the problem for me. This tmpfs is used for something? I mean, shall I mount this after installation?
Thanks you so much again! You made my day!!
Created on 08-09-2017 10:49 AM - edited 08-09-2017 10:50 AM
The mountpoint isn't part of Cloudera, and likely being mounted by other process in your system. The process that controls the tmpfs mount should be able to remount it again, however you will run into the issue again if you hard_stop the agent. The fix is already been committed and will be available in the next Cloudera maintenance release.
Created 08-09-2017 11:55 PM
Created 08-10-2017 12:01 AM
Does not the Michalis' provided solution worked for you?
In this case:
1. Can we see the content of your /etc/hosts file?
2. Can we see the conten of your /etc/hostname file?
3. What are the result of the following commands execution?:
-$ hostname
-$ hostname -f
-$ python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'
Any log errors?
Thanks.
Created 08-10-2017 12:04 AM
[root@bigdata-master cloudera-scm-agent]# hostname
bigdata-master
[root@bigdata-master cloudera-scm-agent]# hostname -f
bigdata-master.lcloud.com
[root@bigdata-master cloudera-scm-agent]# python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'
bigdata-master.lcloud.com 192.168.1.127
[root@bigdata-master cloudera-scm-agent]#
====
ths,i run the command and got this,is this any problem?
Created 08-10-2017 12:13 AM
could you share your /etc/hosts ?
status of your firewall ?
what os are you runining ?