Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Cloudera 5 installation on CentOS : Installation failed. Failed to receive heartbeat from agent

avatar
New Contributor

Hello all !

 

This is my first time in Hadoop / Cloudera world 🙂

 

I installed Cent OS 6.5 in VM-Ware (in top of Windows 7), and installed Cloudera Manager 5.

I configured SSH with private key for Linux user "cloudera", and launched installation using url http://localhost:7180

 

But during installation on one single node (same as CM host), i have this error :

 

Installation failed. Failed to receive heartbeat from agent.

    Ensure that the host's hostname is configured properly.
    Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
    Ensure that ports 9000 and 9001 are free on the host being added.
    Check agent logs in /var/log/cloudera-scm-agent/ on the host being added (some of the logs can be found in the installation details).

 I tried to disable firewall as a test, but same thing.

As hostname, i tried localhost and my IP adress (192.168.241.132), but same thing...

 

Here are the logs from /var/log/cloudera-scm-agent/ :

 

[11/Apr/2014 04:24:19 +0000] 4550 MainThread tmpfs        INFO     Reusing mounted tmpfs at /var/run/cloudera-scm-agent/process
[11/Apr/2014 04:24:20 +0000] 4550 MainThread agent        INFO     Trying to connect to newly launched supervisor (Attempt 1)
[11/Apr/2014 04:24:20 +0000] 4550 MainThread agent        INFO     Successfully connected to supervisor
[11/Apr/2014 04:24:20 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:24:20] ENGINE Bus STARTING
[11/Apr/2014 04:24:20 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:24:20] ENGINE Started monitor thread '_TimeoutMonitor'.
[11/Apr/2014 04:24:21 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:24:21] ENGINE Serving on localhost.localdomain:9000
[11/Apr/2014 04:24:21 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:24:21] ENGINE Bus STARTED
[11/Apr/2014 04:24:21 +0000] 4550 MainThread __init__     INFO     New monitor: (<cmf.monitor.host.HostMonitor object at 0x234de10>,)
[11/Apr/2014 04:24:21 +0000] 4550 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('HostMonitor',)
[11/Apr/2014 04:24:21 +0000] 4550 MainThread agent        WARNING  Setting default socket timeout to 30!
[11/Apr/2014 04:24:21 +0000] 4550 MainThread agent        INFO     Using parcels directory from server provided value: /opt/cloudera/parcels
[11/Apr/2014 04:24:21 +0000] 4550 MainThread parcel       INFO     Agent does create users/groups and apply file permissions
[11/Apr/2014 04:24:21 +0000] 4550 MainThread downloader   INFO     Downloader path: /opt/cloudera/parcel-cache
[11/Apr/2014 04:24:21 +0000] 4550 MainThread parcel_cache INFO     Using /opt/cloudera/parcel-cache for parcel cache
[11/Apr/2014 04:24:21 +0000] 4550 MainThread agent        INFO     Active parcel list updated; recalculating component info.
[11/Apr/2014 04:24:26 +0000] 4550 Monitor-HostMonitor throttling_logger INFO     Using java location: '/usr/java/jdk1.7.0_45-cloudera/bin/java'.
[11/Apr/2014 04:24:26 +0000] 4550 Monitor-HostMonitor throttling_logger WARNING  hostname localhost.localdomain differs from the canonical name localhost
[11/Apr/2014 04:39:58 +0000] 4550 MainThread agent        INFO     Stopping agent...
[11/Apr/2014 04:39:58 +0000] 4550 MainThread agent        INFO     No extant cgroups; unmounting any cgroup roots
[11/Apr/2014 04:39:58 +0000] 4550 MainThread agent        INFO     No processes are being managed; Supervisor will shutdown.
[11/Apr/2014 04:39:58 +0000] 4550 MainThread agent        INFO     Shutting down supervisord, pid 4574
[11/Apr/2014 04:39:59 +0000] 4550 MainThread agent        INFO     waiting for process to terminate...
[11/Apr/2014 04:39:59 +0000] 4550 MainThread agent        INFO     Successfully killed process with pid 4574
[11/Apr/2014 04:39:59 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:39:59] ENGINE Bus STOPPING
[11/Apr/2014 04:39:59 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:39:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('localhost.localdomain', 9000)) shut down
[11/Apr/2014 04:39:59 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:39:59] ENGINE Stopped thread '_TimeoutMonitor'.
[11/Apr/2014 04:39:59 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:39:59] ENGINE Bus STOPPED
[11/Apr/2014 04:39:59 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:39:59] ENGINE Bus STOPPING
[11/Apr/2014 04:39:59 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:39:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('localhost.localdomain', 9000)) already shut down
[11/Apr/2014 04:39:59 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:39:59] ENGINE No thread running for None.
[11/Apr/2014 04:39:59 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:39:59] ENGINE Bus STOPPED
[11/Apr/2014 04:39:59 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:39:59] ENGINE Bus EXITING
[11/Apr/2014 04:39:59 +0000] 4550 MainThread _cplogging   INFO     [11/Apr/2014:04:39:59] ENGINE Bus EXITED
[11/Apr/2014 04:39:59 +0000] 4550 MainThread agent        INFO     Agent exiting; caught signal 15
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     No command line vars
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     Missing database jar: /usr/share/java/mysql-connector-java.jar (normal, if you're not using this database type)
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type)
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     Agent starting as pid 8020 user root(0) group root(0).
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups
[11/Apr/2014 04:40:00 +0000] 8020 MainThread cgroups      INFO     Reusing /var/run/cloudera-scm-agent/cgroups/memory
[11/Apr/2014 04:40:00 +0000] 8020 MainThread cgroups      INFO     Reusing /var/run/cloudera-scm-agent/cgroups/cpu
[11/Apr/2014 04:40:00 +0000] 8020 MainThread cgroups      INFO     Reusing /var/run/cloudera-scm-agent/cgroups/cpuacct
[11/Apr/2014 04:40:00 +0000] 8020 MainThread cgroups      INFO     Reusing /var/run/cloudera-scm-agent/cgroups/blkio
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True}
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     Setting up supervisord event monitor.
[11/Apr/2014 04:40:00 +0000] 8020 MainThread filesystem_map INFO     Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
[11/Apr/2014 04:40:00 +0000] 8020 MainThread filesystem_map INFO     Using timeout of 2.000000
[11/Apr/2014 04:40:00 +0000] 8020 MainThread filesystem_map INFO     Using join timeout of 0.100000
[11/Apr/2014 04:40:00 +0000] 8020 MainThread filesystem_map INFO     Using tolerance of 60.000000
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     Using metrics_url_timeout_seconds of 30.000000
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     Using task_metrics_timeout_seconds of 5.000000
[11/Apr/2014 04:40:00 +0000] 8020 MainThread agent        INFO     Using max_collection_wait_seconds of 10.000000
[11/Apr/2014 04:40:00 +0000] 8020 MainThread metrics      INFO     Importing tasktracker metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json
[11/Apr/2014 04:40:00 +0000] 8020 MainThread dns_names    INFO     Using timeout of 2.000000
[11/Apr/2014 04:40:00 +0000] 8020 MainThread ntp_monitor  INFO     Using timeout of 2.000000
[11/Apr/2014 04:40:00 +0000] 8020 MainThread __init__     INFO     Importing metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json
[11/Apr/2014 04:40:01 +0000] 8020 MainThread agent        INFO     Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_SPARK_HOME': '/usr/lib/spark', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_HUE_HOME': '/usr/lib/hue', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_FLUME_HOME': '/usr/lib/flume-ng', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_LLAMA_HOME': '/usr/lib/llama/'}
[11/Apr/2014 04:40:01 +0000] 8020 MainThread agent        INFO     To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[11/Apr/2014 04:40:01 +0000] 8020 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
[11/Apr/2014 04:40:01 +0000] 8020 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
[11/Apr/2014 04:40:01 +0000] 8020 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
[11/Apr/2014 04:40:01 +0000] 8020 MainThread agent        ERROR    Failed to connect to previous supervisor.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1231, in find_or_start_supervisor
    self.get_supervisor_process_info()
  File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1418, in get_supervisor_process_info
    self.identifier = self.supervisor_client.supervisor.getIdentification()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/xmlrpc.py", line 460, in request
    self.connection.request('POST', handler, request_body, self.headers)
  File "/usr/lib64/python2.6/httplib.py", line 914, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
    self.endheaders()
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
    self._send_output()
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
    self.connect()
  File "/usr/lib64/python2.6/httplib.py", line 720, in connect
    self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
    raise error, msg
error: [Errno 111] Connection refused
[11/Apr/2014 04:40:01 +0000] 8020 MainThread tmpfs        INFO     Reusing mounted tmpfs at /var/run/cloudera-scm-agent/process
[11/Apr/2014 04:40:02 +0000] 8020 MainThread agent        INFO     Trying to connect to newly launched supervisor (Attempt 1)
[11/Apr/2014 04:40:02 +0000] 8020 MainThread agent        INFO     Successfully connected to supervisor
[11/Apr/2014 04:40:02 +0000] 8020 MainThread _cplogging   INFO     [11/Apr/2014:04:40:02] ENGINE Bus STARTING
[11/Apr/2014 04:40:02 +0000] 8020 MainThread _cplogging   INFO     [11/Apr/2014:04:40:02] ENGINE Started monitor thread '_TimeoutMonitor'.
[11/Apr/2014 04:40:02 +0000] 8020 MainThread _cplogging   INFO     [11/Apr/2014:04:40:02] ENGINE Serving on localhost.localdomain:9000
[11/Apr/2014 04:40:02 +0000] 8020 MainThread _cplogging   INFO     [11/Apr/2014:04:40:02] ENGINE Bus STARTED
[11/Apr/2014 04:40:02 +0000] 8020 MainThread __init__     INFO     New monitor: (<cmf.monitor.host.HostMonitor object at 0x1d9be10>,)
[11/Apr/2014 04:40:02 +0000] 8020 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: ('HostMonitor',)
[11/Apr/2014 04:40:02 +0000] 8020 MainThread agent        WARNING  Setting default socket timeout to 30!
[11/Apr/2014 04:40:02 +0000] 8020 MainThread agent        INFO     Using parcels directory from server provided value: /opt/cloudera/parcels
[11/Apr/2014 04:40:02 +0000] 8020 MainThread parcel       INFO     Agent does create users/groups and apply file permissions
[11/Apr/2014 04:40:02 +0000] 8020 MainThread downloader   INFO     Downloader path: /opt/cloudera/parcel-cache
[11/Apr/2014 04:40:02 +0000] 8020 MainThread parcel_cache INFO     Using /opt/cloudera/parcel-cache for parcel cache
[11/Apr/2014 04:40:02 +0000] 8020 MainThread agent        INFO     Active parcel list updated; recalculating component info.
[11/Apr/2014 04:40:07 +0000] 8020 Monitor-HostMonitor throttling_logger INFO     Using java location: '/usr/java/jdk1.7.0_45-cloudera/bin/java'.
[11/Apr/2014 04:40:08 +0000] 8020 Monitor-HostMonitor throttling_logger WARNING  hostname localhost.localdomain differs from the canonical name localhost

 

Any help will be very appreciated 🙂

1 ACCEPTED SOLUTION

avatar
New Contributor
2 REPLIES 2

avatar
New Contributor

avatar
Explorer

There has to be another way. Since Cloudera has yet to create a "QuickStart VM" for CDH5 I'm trying to create it myself, using VMWare and the CentOS 6.5 image to start. But I need to be able to give the VM for others to use so hard coding an IP address in the hosts file is not viable. Indeed in CDH4.4 VMWare image just uses

127.0.0.1 localhost.localdomain localhost.