Installation failed connection refused

[#1]

Hi,

      I am trying to install Hadoop using Cloudera Manager 5.6 and facing the following issue

Installation failed. Failed to receive heartbeat from agent.

• Ensure that the host's hostname is configured properly.
• Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
• Ensure that ports 9000 and 9001 are not in use on the host being added.
• Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).
• If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that/etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.

SCM-AGENT.LOG:

[25/Jul/2016 16:21:58 +0000] 7453 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
[25/Jul/2016 16:21:58 +0000] 7453 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[25/Jul/2016 16:21:58 +0000] 7453 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
[25/Jul/2016 16:21:58 +0000] 7453 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
[25/Jul/2016 16:21:58 +0000] 7453 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
[25/Jul/2016 16:21:58 +0000] 7453 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
[25/Jul/2016 16:21:58 +0000] 7453 MainThread agent ERROR Failed to connect to previous supervisor.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.8.1-py2.6.egg/cmf/agent.py", line 2039, in find_or_start_supervisor
self.get_supervisor_process_info()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.8.1-py2.6.egg/cmf/agent.py", line 2185, in get_supervisor_process_info
self.identifier = self.supervisor_client.supervisor.getIdentification()
File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
verbose=self.__verbose
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/xmlrpc.py", line 460, in request
self.connection.request('POST', handler, request_body, self.headers)
File "/usr/lib64/python2.6/httplib.py", line 914, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
self.endheaders()
File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
self._send_output()
File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
self.send(msg)
File "/usr/lib64/python2.6/httplib.py", line 739, in send
self.connect()
File "/usr/lib64/python2.6/httplib.py", line 720, in connect
self.timeout)
File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
raise error, msg
error: [Errno 111] Connection refused
[25/Jul/2016 16:21:58 +0000] 7453 MainThread tmpfs INFO Reusing mounted tmpfs at /var/run/cloudera-scm-agent/process
[25/Jul/2016 16:21:58 +0000] 7453 MainThread agent INFO Deleting stale supervisor include /var/run/cloudera-scm-agent/supervisor/include/flood.conf.
[25/Jul/2016 16:21:59 +0000] 7453 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 1)
[25/Jul/2016 16:21:59 +0000] 7453 MainThread agent INFO Supervisor version: 3.0
[25/Jul/2016 16:21:59 +0000] 7453 MainThread agent INFO Successfully connected to supervisor
[25/Jul/2016 16:21:59 +0000] 7453 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes.
[25/Jul/2016 16:21:59 +0000] 7453 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes.
[25/Jul/2016 16:21:59 +0000] 7453 MainThread _cplogging INFO [25/Jul/2016:16:21:59] ENGINE Bus STARTING
[25/Jul/2016 16:21:59 +0000] 7453 MainThread _cplogging INFO [25/Jul/2016:16:21:59] ENGINE Started monitor thread '_TimeoutMonitor'.

[25/Jul/2016 16:22:00 +0000] 7453 MainThread _cplogging INFO [25/Jul/2016:16:22:00] ENGINE Serving on cdh03.ts.verticacorp.com:9000
[25/Jul/2016 16:22:00 +0000] 7453 MainThread _cplogging INFO [25/Jul/2016:16:22:00] ENGINE Bus STARTED
[25/Jul/2016 16:22:00 +0000] 7453 MainThread __init__ INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x256eb10>,)
[25/Jul/2016 16:22:00 +0000] 7453 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('HostMonitor',)
[25/Jul/2016 16:22:00 +0000] 7453 MainThread agent INFO Setting default socket timeout to 30
[25/Jul/2016 16:22:00 +0000] 7453 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.07 min:0.07 mean:0.07 max:0.07 LIFE_MAX:0.07
[25/Jul/2016 16:22:00 +0000] 7453 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels
[25/Jul/2016 16:22:00 +0000] 7453 MainThread parcel INFO Agent does create users/groups and apply file permissions
[25/Jul/2016 16:22:00 +0000] 7453 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
[25/Jul/2016 16:22:00 +0000] 7453 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
[25/Jul/2016 16:22:00 +0000] 7453 MainThread agent INFO Flood daemon (re)start attempt
[25/Jul/2016 16:22:00 +0000] 7453 MainThread agent INFO Triggering supervisord update.
[25/Jul/2016 16:22:00 +0000] 7453 MainThread downloader ERROR Failed rack peer update: [Errno 111] Connection refused

[#2]

Do you check the /etc/hosts of the machine?

[#3]

Can you reach the CM host on port 7182? I'd check the CM server logs too as they may have more info on why the connection was refused, if it is able to reach it but there is still an issue.

[#4]

Hello,

You mention that you are installing 5.6, but, based on the fact that the error is occurring in code that does not exist in version 5.6 and the exception shows "cm5.8.1", it seems you are actually installing with Cloudera Manager 5.8.1.

The fact that you are seeing the following error:

[25/Jul/2016 16:22:00 +0000] 7453 MainThread downloader ERROR Failed rack peer update: [Errno 111] Connection refused

Means that when the agent attempts to connet to its peers on port 7191, it cannot connect.  This could be due to various reasons including firewall, routing, maybe a transient failure in one of the agents.

The error is occurring during the processing of the Cloudera Manager heartbeat response that includes a list of "peers" that the agent can use to download parcels from.

If you have a small number of hosts, and cannot resolve the port/connection issue, you can revert to the old method of parcel download which would only download from Cloudera Manager (rather than leveraging the peer download feature).

To do so:

* in Cloudera Manager, choose "All Hosts" from the "Hosts" tab. 

* Click the Configuration button on the right of the page.

* Search for "P2P Parcel Distribution Port"

* Set "P2P Parcel Distribution Port" to "0"

* Save

I believe you need to restart the agents with "service cloudera-scm-agent restart" in order for them to pick up on the change.

After doing that, you should be able to proceed.

[#5]

Hello,

   Based on your advise to set "P2P Parcel Distribution Port" to "0" cloudera manager I am able to resove the Rack Peer update error  . But still installation is failing saying "Failed to receive heartbeat from agent" . Agent is shutting down by itsef. I am not able to find any error in the log too . Anyone facing this issue. Please help me. I am using CM 5.9.1. Thanks in advance!!!!!

[#6]

did you fixed it?