Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera Manager cannot detect heartbeat from installed agents

Cloudera Manager cannot detect heartbeat from installed agents

New Contributor

I've setup 2 ec2 instances and I'm trying to install cloudera manager on these boxes.

 

The installation proceeds with no errors until it tries to detect heartbeat from the installed agents.

 

At this point, i get the following message on the UI: Installation failed.

 

Installation failed. Failed to receive heartbeat from agent.

  • Ensure that the host's hostname is configured properly.
  • Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
  • Ensure that ports 9000 and 9001 are free on the host being added.
  • Check agent logs in /var/log/cloudera-scm-agent/ on the host being added (some of the logs can be found in the installation details).

 

And the following trace in the logs:

 

 

/tmp/scm_prepare_node.CDSWQ8NL
using SSH_CLIENT to get the SCM hostname: <MY-IP> 51468 22
opening logging file descriptor
Starting installation script...
Acquiring installation lock...
BEGIN flock 4
END (0)
Detecting root privileges...
effective UID is 0
Detecting distribution...
BEGIN grep 'Ubuntu' /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"
END (0)
BEGIN grep DISTRIB_CODENAME /etc/lsb-release
END (0)
DISTRIB_CODENAME=precise
BEGIN echo DISTRIB_CODENAME=precise | cut -d = -f 2
END (0)
precise
Detecting Cloudera Manager Server...
BEGIN host -t PTR <MY-IP>
Host<MY-IP>.in-addr.arpa not found: 5(REFUSED)
END (1)
BEGIN which python
END (0)
BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' <MY-IP> 7182
/usr/bin/python
END (0)
BEGIN which wget
END (0)
/usr/bin/wget
BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo
ec2-54-188-75-250.us-west-2.compute.amazonaws.com
END (0)
BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/meta-data/instance-id && /bin/echo
i-5770c95d
END (0)
Performing cloud-specific configuration...
BEGIN df -h | grep -o -E "^/dev/[^[:space:]]*"
/dev/xvda1
/dev/xvdb
END (0)
BEGIN awk 'FNR > 2 {print $NF}' /proc/partitions
END (0)
xvda1
xvdb
Installing package repositories...
validating format of repository file /tmp/scm_prepare_node.CDSWQ8NL/repos/ubuntu_precise/cloudera-manager.list
installing repository file /tmp/scm_prepare_node.CDSWQ8NL/repos/ubuntu_precise/cloudera-manager.list
repository file /tmp/scm_prepare_node.CDSWQ8NL/repos/ubuntu_precise/cloudera-manager.list installed
BEGIN apt-key add /tmp/scm_prepare_node.CDSWQ8NL/archive.key
OK
END (0)
Refreshing package metadata...
BEGIN apt-get update
Err http://archive.cloudera.com precise-cm5 Release.gpg
Temporary failure resolving 'archive.cloudera.com'
Err http://us-west-2.ec2.archive.ubuntu.com precise Release.gpg
Temporary failure resolving 'us-west-2.ec2.archive.ubuntu.com'
Err http://security.ubuntu.com precise-security Release.gpg
Temporary failure resolving 'security.ubuntu.com'
Err http://archive.cloudera.com precise-cm5.2.0 Release.gpg
Temporary failure resolving 'archive.cloudera.com'
Err http://us-west-2.ec2.archive.ubuntu.com precise-updates Release.gpg
Temporary failure resolving 'us-west-2.ec2.archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://us-west-2.ec2.archive.ubuntu.com/ubuntu/dists/precise/Release.gpg Temporary failure resolving 'us-west-2.ec2.archive.ubuntu.com'
W: Failed to fetch http://us-west-2.ec2.archive.ubuntu.com/ubuntu/dists/precise-updates/Release.gpg Temporary failure resolving 'us-west-2.ec2.archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/precise-security/Release.gpg Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5/Release.gpg Temporary failure resolving 'archive.cloudera.com'
W: Failed to fetch http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.2.0/Release.gpg Temporary failure resolving 'archive.cloudera.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
END (0)
BEGIN apt-get update
Err http://security.ubuntu.com precise-security Release.gpg
Temporary failure resolving 'security.ubuntu.com'
Err http://archive.cloudera.com precise-cm5 Release.gpg
Temporary failure resolving 'archive.cloudera.com'
Err http://us-west-2.ec2.archive.ubuntu.com precise Release.gpg
Temporary failure resolving 'us-west-2.ec2.archive.ubuntu.com'
Err http://us-west-2.ec2.archive.ubuntu.com precise-updates Release.gpg
Temporary failure resolving 'us-west-2.ec2.archive.ubuntu.com'
Err http://archive.cloudera.com precise-cm5.2.0 Release.gpg
Temporary failure resolving 'archive.cloudera.com'
Reading package lists...
W: Failed to fetch http://us-west-2.ec2.archive.ubuntu.com/ubuntu/dists/precise/Release.gpg Temporary failure resolving 'us-west-2.ec2.archive.ubuntu.com'
W: Failed to fetch http://us-west-2.ec2.archive.ubuntu.com/ubuntu/dists/precise-updates/Release.gpg Temporary failure resolving 'us-west-2.ec2.archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/precise-security/Release.gpg Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5/Release.gpg Temporary failure resolving 'archive.cloudera.com'
W: Failed to fetch http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.2.0/Release.gpg Temporary failure resolving 'archive.cloudera.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
END (0)
Installing oracle-j2sdk1.6 package...
BEGIN dpkg -l oracle-j2sdk1.6 | grep -E '^ii[[:space:]]*oracle-j2sdk1.6[[:space:]]*'
ii oracle-j2sdk1.6 1.6.0+update31 Java(TM) JDK, Standard Edition, Oracle(TM)
END (0)
BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep oracle-j2sdk1.6
oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons
END (0)
BEGIN apt-cache show oracle-j2sdk1.6
Package: oracle-j2sdk1.6
Version: 1.6.0+update31
Architecture: amd64
Maintainer: Debian Java Maintainers <pkg-java-maintainers@lists.alioth.debian.org>
Installed-Size: 224994
Depends: libasound2 (>= 1.0.16), libc6 (>= 2.2.5), libx11-6, libxext6, libxi6, libxt6, libxtst6
Recommends: netbase, libgtk1.2, libstdc++5
Replaces: oracle-j2sdk1.6debian
Provides: j2re1.6, j2sdk1.6, java-browser-plugin, java-compiler, java-runtime, java-runtime-headless, java-sdk, java-virtual-machine, java2-compiler, java2-runtime, java2-runtime-headless, java2-sdk
Priority: optional
Section: non-free/devel
Filename: pool/contrib/o/oracle-j2sdk1.6/oracle-j2sdk1.6_1.6.0+update31_amd64.deb
Size: 96896162
SHA256: e3af578020ca3d595fbf1da7f006dde783fa10ace7e147ea135e5d09a8ad32b9
SHA1: 4b4cd040b2b71f1b754180d89c8e69f5940d27a5
MD5sum: 183149f3ddc42fcc03e137bea41cf85e
Description: Java(TM) JDK, Standard Edition, Oracle(TM)
The Java(TM) 2 SDK is a development environment for building
applications, applets, and components that can be deployed on the
Java(TM) platform.
.
The Java(TM) 2 SDK software includes tools useful for developing and
testing programs written in the Java programming language and running
on the Java platform. These tools are designed to be used from the
command line. Except for appletviewer, these tools do not provide a
graphical user interface.
.
This package has been automatically created with java-package (0.50).
END (0)
BEGIN apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install oracle-j2sdk1.6
Reading package lists...
Building dependency tree...
Reading state information...
oracle-j2sdk1.6 is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 32 not upgraded.
END (0)
remote package oracle-j2sdk1.6 installed
Installing oracle-j2sdk1.7 package...
BEGIN dpkg -l oracle-j2sdk1.7 | grep -E '^ii[[:space:]]*oracle-j2sdk1.7[[:space:]]*'
ii oracle-j2sdk1.7 1.7.0+update67-1 no description given
END (0)
BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep oracle-j2sdk1.7
END (0)
oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons
BEGIN apt-cache show oracle-j2sdk1.7
Package: oracle-j2sdk1.7
Version: 1.7.0+update67-1
License: unknown
Vendor: root@rpmslave.ent.cloudera.com
Architecture: amd64
Maintainer: <scm-users@cloudera.org>
Installed-Size: 285587
Provides: java
Homepage: http://example.com/no-uri-given
Priority: extra
Section: default
Filename: pool/contrib/o/oracle-j2sdk1.7/oracle-j2sdk1.7_1.7.0+update67-1_amd64.deb
Size: 142448292
SHA256: b998f4aa8ff8e58ee1d7d55e409330034c8963dbdab07f164fe7baf2025f863e
SHA1: 2f03993be2d96a6b2abf2259e77bd21ef1a2bbe4
MD5sum: f7e1bc66703c986be480001ddd087135
Description: no description given
END (0)
BEGIN apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install oracle-j2sdk1.7
Reading package lists...
Building dependency tree...
Reading state information...
oracle-j2sdk1.7 is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 32 not upgraded.
END (0)
remote package oracle-j2sdk1.7 installed
Installing cloudera-manager-agent package...
BEGIN dpkg -l cloudera-manager-agent | grep -E '^ii[[:space:]]*cloudera-manager-agent[[:space:]]*'
ii cloudera-manager-agent 5.2.0-1.cm520.p0.60~precise-cm5.2.0 The Cloudera Manager Agent
END (0)
BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-agent
oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons
END (0)
BEGIN apt-cache show cloudera-manager-agent
Package: cloudera-manager-agent
Source: enterprise
Version: 5.2.0-1.cm520.p0.60~precise-cm5.2.0
Architecture: amd64
Maintainer: Cloudera Inc. <https://issues.cloudera.org>
Installed-Size: 22702
Depends: libc6 (>= 2.15), libcomerr2 (>= 1.01), libgcc1 (>= 1:4.1.1), libgssapi-krb5-2 (>= 1.10+dfsg~), libkrb5-3 (>= 1.6.dfsg.2), libssl1.0.0 (>= 1.0.0), zlib1g (>= 1:1.2.0), lsb-base, psmisc, bash, libsasl2-modules, libsasl2-modules-gssapi-mit, libxslt1.1, libsqlite3-0, libfuse2, fuse-utils | fuse, rpcbind, cloudera-manager-daemons (= 5.2.0-1.cm520.p0.60~precise-cm5.2.0)
Homepage: http://www.cloudera.com
Priority: extra
Section: misc
Filename: pool/contrib/e/enterprise/cloudera-manager-agent_5.2.0-1.cm520.p0.60~precise-cm5.2.0_amd64.deb
Size: 5894434
SHA256: 742b2103f99a79c3e0df670cd215d5b3e259dad4d22c15018cc630de1d198ec5
SHA1: 3223c00bfb5cd7e9342f612bdad4eab3463fa9d7
MD5sum: bd64d5051c028616e3cf7d324a947c82
Description: The Cloudera Manager Agent
The Agent is deployed to machines running services managed by Cloudera Manager.
END (0)
Version: 5.2.0-1.cm520.p0.60~precise-cm5.2.0
BEGIN apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-agent
Reading package lists...
Building dependency tree...
Reading state information...
cloudera-manager-agent is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 32 not upgraded.
END (0)
remote package cloudera-manager-agent installed
Installing cloudera-manager-daemons package...
BEGIN dpkg -l cloudera-manager-daemons | grep -E '^ii[[:space:]]*cloudera-manager-daemons[[:space:]]*'
ii cloudera-manager-daemons 5.2.0-1.cm520.p0.60~precise-cm5.2.0 Provides daemons for monitoring Hadoop and related tools.
END (0)
BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-daemons
END (0)
oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons
BEGIN apt-cache show cloudera-manager-daemons
Package: cloudera-manager-daemons
Source: enterprise
Version: 5.2.0-1.cm520.p0.60~precise-cm5.2.0
Architecture: all
Maintainer: Cloudera Inc. <https://issues.cloudera.org>
Installed-Size: 595711
Homepage: http://www.cloudera.com
Priority: extra
Section: misc
Filename: pool/contrib/e/enterprise/cloudera-manager-daemons_5.2.0-1.cm520.p0.60~precise-cm5.2.0_all.deb
Size: 442781200
SHA256: 70c1c2ec2c48ac9dfa16b0d4a8a9d23c0fbf00b86e9aead06ec38045cd5237ff
SHA1: a6850a2b70177c6fc02750313f20f738a9692b3c
MD5sum: ba7ee3df78f5645824c6237d64089204
Description: Provides daemons for monitoring Hadoop and related tools.
Provides daemons for monitoring Hadoop and related tools.
END (0)
Version: 5.2.0-1.cm520.p0.60~precise-cm5.2.0
BEGIN apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-daemons
Reading package lists...
Building dependency tree...
Reading state information...
cloudera-manager-daemons is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 32 not upgraded.
END (0)
remote package cloudera-manager-daemons installed
Installing Unlimited Strength Encryption policy files.
Installation not requested. Step will be skipped.
Configuring Cloudera Manager Agent...
BEGIN sed -e 's/\(CMF_AGENT_ARGS=\).*/\1"--host_id i-5770c95d"/' -i /etc/default/cloudera-scm-agent
END (0)
BEGIN grep server_host=<MY-IP> /etc/cloudera-scm-agent/config.ini
END (0)
scm agent is already configured
Starting Cloudera Manager Agent...
server_host=<MY-IP>
BEGIN /usr/sbin/service cloudera-scm-agent status | grep running
END (1)
BEGIN /usr/sbin/service cloudera-scm-agent start
Starting cloudera-scm-agent: * Couldn't start cloudera-scm-agent
END (0)
agent logs:
BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/'
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO SCM Agent Version: 5.2.0
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Adding env vars that start with CMF_AGENT_
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
>>R2D2: 2 || || 6 || ('<MY-IP>', 9000) || 1>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO SCM Agent Version: 5.2.0
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Adding env vars that start with CMF_AGENT_
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
>>R2D2: 2 || || 6 || ('<MY-IP>', 9000) || 1END (0)
BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/'
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Agent is running on a public cloud. Public hostname: ec2-54-188-75-250.us-west-2.compute.amazonaws.com
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/cgroups
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Found cgroups subsystem: cpu
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Found cgroups subsystem: cpuacct
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Found cgroups subsystem: memory
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Found cgroups subsystem: blkio
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/memory
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpu
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpuacct
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/blkio
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True}
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Setting up supervisord event monitor.
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread filesystem_map INFO Using timeout of 2.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread filesystem_map INFO Using join timeout of 0.100000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread filesystem_map INFO Using tolerance of 60.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4']
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Using max_collection_wait_seconds of 10.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib/cmf/agent/src/cmf/monitor/tasktracker/schema.json
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread dns_names INFO Using timeout of 2.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread ntp_monitor INFO Using timeout of 2.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread __init__ INFO Importing metric schema from file /usr/lib/cmf/agent/src/cmf/monitor/schema.json
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/process
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Supervisor version: 3.0
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Connecting to previous supervisor: agent-27477-1417109669.
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes.
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes.
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Bus STARTING
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Started monitor thread '_TimeoutMonitor'.
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging ERROR [01/Dec/2014:06:24:54] ENGINE Error in HTTP server: shutting down
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/CherryPy-3.2.2-py2.7.egg/cherrypy/process/servers.py", line 187, in _start_http_thread
>> self.httpserver.start()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/CherryPy-3.2.2-py2.7.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1827, in start
>> raise socket.error(msg)
>>error: No socket could be created on ('<MY-PUBLIC-FACING-HOST-HOSTNAME>', 9000) -- [Errno 99] Cannot assign requested address
>>
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Bus STOPPING
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('<MY-PUBLIC-FACING-HOST-HOSTNAME>', 9000)) already shut down
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Stopped thread '_TimeoutMonitor'.
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Bus STOPPED
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Bus EXITING
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Bus EXITED
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Agent is running on a public cloud. Public hostname: ec2-54-188-75-250.us-west-2.compute.amazonaws.com
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/cgroups
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Found cgroups subsystem: cpu
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Found cgroups subsystem: cpuacct
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Found cgroups subsystem: memory
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Found cgroups subsystem: blkio
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/memory
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpu
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpuacct
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/blkio
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True}
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Setting up supervisord event monitor.
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread filesystem_map INFO Using timeout of 2.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread filesystem_map INFO Using join timeout of 0.100000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread filesystem_map INFO Using tolerance of 60.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4']
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Using max_collection_wait_seconds of 10.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib/cmf/agent/src/cmf/monitor/tasktracker/schema.json
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread dns_names INFO Using timeout of 2.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread ntp_monitor INFO Using timeout of 2.000000
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread __init__ INFO Importing metric schema from file /usr/lib/cmf/agent/src/cmf/monitor/schema.json
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/process
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Supervisor version: 3.0
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread agent INFO Connecting to previous supervisor: agent-27477-1417109669.
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes.
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes.
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Bus STARTING
>>[01/Dec/2014 06:24:54 +0000] 29172 MainThread _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Started monitor thread '_TimeoutMonitor'.
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging ERROR [01/Dec/2014:06:24:54] ENGINE Error in HTTP server: shutting down
>>Traceback (most recent call last):
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/CherryPy-3.2.2-py2.7.egg/cherrypy/process/servers.py", line 187, in _start_http_thread
>> self.httpserver.start()
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/CherryPy-3.2.2-py2.7.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1827, in start
>> raise socket.error(msg)
>>error: No socket could be created on ('<MY-PUBLIC-FACING-HOST-HOSTNAME>', 9000) -- [Errno 99] Cannot assign requested address
>>
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Bus STOPPING
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('<MY-PUBLIC-FACING-HOST-HOSTNAME>', 9000)) already shut down
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Stopped thread '_TimeoutMonitor'.
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Bus STOPPED
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Bus EXITING
>>[01/Dec/2014 06:24:54 +0000] 29172 HTTPServer Thread-2 _cplogging INFO [01/Dec/2014:06:24:54] ENGINE Bus EXITED
END (0)
end of agent logs.
scm agent started

 

Running this: python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'

 

returns FQDN and IP correctly.

 

I've also opened up the required ports on ec2.

 

I'd appreciate any help as I'm at a loss as to what's wrong. Thanks.

1 REPLY 1
Highlighted

Re: Cloudera Manager cannot detect heartbeat from installed agents

Contributor

did you disable iptables and selinux? those can cause issues. Also did you add the nodes to the /etc/hosts? I noticed at times you have to do to be able to find each other correctly.

Don't have an account?
Coming from Hortonworks? Activate your account here