Reply
Explorer
Posts: 12
Registered: ‎09-25-2017

Cloudera Manager distribution error

I am trying to install cloudera manager on single host in psuedo distribution. The OS is ubuntu 16.04. I am getting the error at distribution phase of installation. It's showing at first as connection reset by peer and then connection refused.

 

This is the log file

 

[04/Oct/2017 20:41:27 +0000] 5147 MainThread agent        INFO     Created /run/cloudera-scm-agent/process/1-cluster-host-inspector
[04/Oct/2017 20:41:27 +0000] 5147 MainThread agent        INFO     Chowning /run/cloudera-scm-agent/process/1-cluster-host-inspector to root (0) root (0)
[04/Oct/2017 20:41:27 +0000] 5147 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/process/1-cluster-host-inspector to 0751
[04/Oct/2017 20:41:27 +0000] 5147 MainThread agent        INFO     Created /run/cloudera-scm-agent/process/1-cluster-host-inspector/logs
[04/Oct/2017 20:41:27 +0000] 5147 MainThread agent        INFO     Chowning /run/cloudera-scm-agent/process/1-cluster-host-inspector/logs to root (0) root (0)
[04/Oct/2017 20:41:27 +0000] 5147 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/process/1-cluster-host-inspector/logs to 0751
[04/Oct/2017 20:41:27 +0000] 5147 MainThread process      INFO     [1-cluster-host-inspector] Refreshing process files: None
[04/Oct/2017 20:41:27 +0000] 5147 MainThread parcel       INFO     prepare_environment begin: {}, [], []
[04/Oct/2017 20:41:27 +0000] 5147 MainThread parcel       INFO     No parcels activated for use
[04/Oct/2017 20:41:27 +0000] 5147 MainThread __init__     INFO     Extracted 1 files and 0 dirs to /run/cloudera-scm-agent/process/1-cluster-host-inspector.
[04/Oct/2017 20:41:27 +0000] 5147 MainThread process      INFO     [1-cluster-host-inspector] Launching process. one-off True, command mgmt/mgmt.sh, args [u'inspector', u'input.json', u'output.json', u'DEFAULT']
[04/Oct/2017 20:41:27 +0000] 5147 MainThread agent        INFO     Triggering supervisord update.
[04/Oct/2017 20:41:27 +0000] 5147 MainThread process      INFO     Begin audit plugin refresh
[04/Oct/2017 20:41:27 +0000] 5147 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for cluster-host-inspector with pipelines []
[04/Oct/2017 20:41:27 +0000] 5147 MainThread process      INFO     Begin metadata plugin refresh
[04/Oct/2017 20:41:27 +0000] 5147 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for cluster-host-inspector with pipelines []
[04/Oct/2017 20:41:27 +0000] 5147 MainThread process      INFO     Not creating a monitor for 1-cluster-host-inspector: should_monitor returns false
[04/Oct/2017 20:41:27 +0000] 5147 MainThread process      INFO     Daemon refresh complete for process 1-cluster-host-inspector.
[04/Oct/2017 20:41:30 +0000] 5147 CP Server Thread-8 _cplogging   INFO     127.0.0.1 - - [04/Oct/2017:20:41:30] "GET /process/1-cluster-host-inspector/files/inspector HTTP/1.1" 200 2252 "" "Java/1.7.0_67"
[04/Oct/2017 20:41:31 +0000] 5147 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for cluster-host-inspector with pipelines []
[04/Oct/2017 20:41:31 +0000] 5147 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cluster-host-inspector (log dir: None)
[04/Oct/2017 20:41:31 +0000] 5147 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for cluster-host-inspector with pipelines []
[04/Oct/2017 20:41:31 +0000] 5147 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cluster-host-inspector (log dir: None)
[04/Oct/2017 20:41:45 +0000] 5147 MainThread process      INFO     [1-cluster-host-inspector] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
[04/Oct/2017 20:41:45 +0000] 5147 MainThread process      INFO     Deactivating process 1-cluster-host-inspector
[04/Oct/2017 20:41:46 +0000] 5147 Audit-Plugin navigator_plugin INFO     stopping Audit Plugin for cluster-host-inspector with pipelines []
[04/Oct/2017 20:41:46 +0000] 5147 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cluster-host-inspector (log dir: None)
[04/Oct/2017 20:41:46 +0000] 5147 Metadata-Plugin navigator_plugin INFO     stopping Metadata Plugin for cluster-host-inspector with pipelines []
[04/Oct/2017 20:41:46 +0000] 5147 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for cluster-host-inspector (log dir: None)
[04/Oct/2017 20:44:50 +0000] 5147 CP Server Thread-9 _cplogging   INFO     127.0.0.1 - - [04/Oct/2017:20:44:50] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[04/Oct/2017 20:44:50 +0000] 5147 MainThread util         INFO     Using generic audit plugin for process zookeeper-init
[04/Oct/2017 20:44:50 +0000] 5147 MainThread util         INFO     Creating metadata plugin for process zookeeper-init
[04/Oct/2017 20:44:50 +0000] 5147 MainThread util         INFO     Using specific metadata plugin for process zookeeper-init
[04/Oct/2017 20:44:50 +0000] 5147 MainThread util         INFO     Using generic metadata plugin for process zookeeper-init
[04/Oct/2017 20:44:50 +0000] 5147 MainThread agent        INFO     [2-zookeeper-init] Instantiating process
[04/Oct/2017 20:44:50 +0000] 5147 MainThread process      INFO     [2-zookeeper-init] Updating process: True {}
[04/Oct/2017 20:44:50 +0000] 5147 MainThread process      INFO     First time to activate the process [2-zookeeper-init].
[04/Oct/2017 20:44:50 +0000] 5147 MainThread agent        INFO     Created /run/cloudera-scm-agent/process/2-zookeeper-init
[04/Oct/2017 20:44:50 +0000] 5147 MainThread agent        INFO     Chowning /run/cloudera-scm-agent/process/2-zookeeper-init to zookeeper (996) zookeeper (995)
[04/Oct/2017 20:44:50 +0000] 5147 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/process/2-zookeeper-init to 0751
[04/Oct/2017 20:44:50 +0000] 5147 MainThread agent        INFO     Created /run/cloudera-scm-agent/process/2-zookeeper-init/logs
[04/Oct/2017 20:44:50 +0000] 5147 MainThread agent        INFO     Chowning /run/cloudera-scm-agent/process/2-zookeeper-init/logs to zookeeper (996) zookeeper (995)
[04/Oct/2017 20:44:50 +0000] 5147 MainThread agent        INFO     Chmod'ing /run/cloudera-scm-agent/process/2-zookeeper-init/logs to 0751
[04/Oct/2017 20:44:50 +0000] 5147 MainThread process      INFO     [2-zookeeper-init] Refreshing process files: None
[04/Oct/2017 20:44:50 +0000] 5147 MainThread parcel       INFO     prepare_environment begin: {u'CDH': u'5.12.1-1.cdh5.12.1.p0.3'}, [u'cdh'], [u'cdh-plugin', u'zookeeper-plugin']
[04/Oct/2017 20:44:50 +0000] 5147 MainThread parcel       INFO     The following requested parcels are not available: {}
[04/Oct/2017 20:44:50 +0000] 5147 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'sentry', 'solr', 'spark'] for parcel CDH
[04/Oct/2017 20:44:50 +0000] 5147 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.12.1-1.cdh5.12.1.p0.3'}
[04/Oct/2017 20:44:50 +0000] 5147 MainThread __init__     INFO     Extracted 10 files and 0 dirs to /run/cloudera-scm-agent/process/2-zookeeper-init.
[04/Oct/2017 20:44:50 +0000] 5147 MainThread process      INFO     [2-zookeeper-init] Evaluating resource: {u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': True, u'rlimits': None, u'install': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/log/zookeeper/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'zookeeper', u'user': u'zookeeper', u'mode': 493}, u'cpu': None, u'contents': None}
[04/Oct/2017 20:44:50 +0000] 5147 MainThread os_ops       INFO     Created directory /var/log/zookeeper
[04/Oct/2017 20:44:50 +0000] 5147 MainThread os_ops       INFO     Created directory /var/log/zookeeper/stacks
[04/Oct/2017 20:44:50 +0000] 5147 MainThread process      INFO     reading limits: {u'limit_memlock': None, u'limit_fds': None}
[04/Oct/2017 20:44:50 +0000] 5147 MainThread process      INFO     [2-zookeeper-init] Launching process. one-off True, command zookeeper/zk.sh, args [u'init', u'--myid=1', u'ignoreExists']
[04/Oct/2017 20:44:50 +0000] 5147 MainThread agent        INFO     Triggering supervisord update.
[04/Oct/2017 20:44:50 +0000] 5147 MainThread process      INFO     Begin audit plugin refresh
[04/Oct/2017 20:44:50 +0000] 5147 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for zookeeper-init with pipelines []
[04/Oct/2017 20:44:50 +0000] 5147 MainThread process      INFO     Begin metadata plugin refresh
[04/Oct/2017 20:44:50 +0000] 5147 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for zookeeper-init with pipelines []
[04/Oct/2017 20:44:50 +0000] 5147 MainThread process      INFO     Not creating a monitor for 2-zookeeper-init: should_monitor returns false
[04/Oct/2017 20:44:50 +0000] 5147 MainThread process      INFO     Daemon refresh complete for process 2-zookeeper-init.
[04/Oct/2017 20:44:51 +0000] 5147 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for zookeeper-init with pipelines []
[04/Oct/2017 20:44:51 +0000] 5147 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for zookeeper-init (log dir: None)
[04/Oct/2017 20:44:51 +0000] 5147 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for zookeeper-init with pipelines []
[04/Oct/2017 20:44:51 +0000] 5147 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for zookeeper-init (log dir: None)
[04/Oct/2017 20:45:07 +0000] 5147 MainThread process      INFO     [2-zookeeper-init] Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
[04/Oct/2017 20:45:07 +0000] 5147 MainThread process      INFO     Deactivating process 2-zookeeper-init
[04/Oct/2017 20:45:11 +0000] 5147 Audit-Plugin navigator_plugin INFO     stopping Audit Plugin for zookeeper-init with pipelines []
[04/Oct/2017 20:45:11 +0000] 5147 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for zookeeper-init (log dir: None)
[04/Oct/2017 20:45:11 +0000] 5147 Metadata-Plugin navigator_plugin INFO     stopping Metadata Plugin for zookeeper-init with pipelines []
[04/Oct/2017 20:45:11 +0000] 5147 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for zookeeper-init (log dir: None)
[04/Oct/2017 20:45:22 +0000] 5147 MainThread firehoses    INFO     Reporting interval updated: 5.0 -> 60
[04/Oct/2017 20:46:20 +0000] 5147 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the SERVICEMONITOR.
[04/Oct/2017 20:46:20 +0000] 5147 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the HOSTMONITOR.
[04/Oct/2017 20:46:20 +0000] 5147 MonitorDaemon-Reporter throttling_logger ERROR    Error sending messages to firehose: mgmt-HOSTMONITOR-58d97c64d44220f47b91dc6ed632f373
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.1-py2.7.egg/cmf/monitor/firehose.py", line 116, in _send
    self._port)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__
    self.conn.connect()
  File "/usr/lib/python2.7/httplib.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 575, in create_connection
    raise err
error: [Errno 111] Connection refused
[04/Oct/2017 20:50:07 +0000] 5147 MainThread heartbeat_tracker INFO     HB stats (seconds): num:46 LIFE_MIN:0.01 min:0.00 mean:0.03 max:0.16 LIFE_MAX:0.07
[04/Oct/2017 20:51:22 +0000] 5147 MainThread throttling_logger INFO     (15 skipped) Identified java component java6 with full version JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)  for requested version 6.
[04/Oct/2017 20:51:22 +0000] 5147 MainThread throttling_logger INFO     (15 skipped) Identified java component java7 with full version JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)  for requested version 7.
[04/Oct/2017 20:57:20 +0000] 5147 MonitorDaemon-Reporter throttling_logger ERROR    (10 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-58d97c64d44220f47b91dc6ed632f373
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.1-py2.7.egg/cmf/monitor/firehose.py", line 116, in _send
    self._port)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__
    self.conn.connect()
  File "/usr/lib/python2.7/httplib.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 575, in create_connection
    raise err
error: [Errno 111] Connection refused
[04/Oct/2017 21:00:07 +0000] 5147 MainThread heartbeat_tracker INFO     HB stats (seconds): num:40 LIFE_MIN:0.01 min:0.01 mean:0.02 max:0.03 LIFE_MAX:0.07
[04/Oct/2017 21:07:20 +0000] 5147 MonitorDaemon-Reporter throttling_logger ERROR    (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-58d97c64d44220f47b91dc6ed632f373
[04/Oct/2017 21:10:08 +0000] 5147 MainThread heartbeat_tracker INFO     HB stats (seconds): num:40 LIFE_MIN:0.01 min:0.01 mean:0.02 max:0.03 LIFE_MAX:0.07
[04/Oct/2017 21:17:20 +0000] 5147 MonitorDaemon-Reporter throttling_logger ERROR    (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-58d97c64d44220f47b91dc6ed632f373
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.1-py2.7.egg/cmf/monitor/firehose.py", line 116, in _send
    self._port)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__
    self.conn.connect()
  File "/usr/lib/python2.7/httplib.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 575, in create_connection
    raise err
error: [Errno 111] Connection refused
[04/Oct/2017 21:20:08 +0000] 5147 MainThread heartbeat_tracker INFO     HB stats (seconds): num:40 LIFE_MIN:0.01 min:0.01 mean:0.02 max:0.03 LIFE_MAX:0.07
[04/Oct/2017 21:21:23 +0000] 5147 MainThread throttling_logger INFO     (14 skipped) Identified java component java6 with full version JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)  for requested version 6.
[04/Oct/2017 21:21:23 +0000] 5147 MainThread throttling_logger INFO     (14 skipped) Identified java component java7 with full version JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)  for requested version 7.
[04/Oct/2017 21:27:20 +0000] 5147 MonitorDaemon-Reporter throttling_logger ERROR    (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-58d97c64d44220f47b91dc6ed632f373
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.1-py2.7.egg/cmf/monitor/firehose.py", line 116, in _send
    self._port)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__
    self.conn.connect()
  File "/usr/lib/python2.7/httplib.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 575, in create_connection
    raise err
error: [Errno 111] Connection refused
[04/Oct/2017 21:30:09 +0000] 5147 MainThread heartbeat_tracker INFO     HB stats (seconds): num:40 LIFE_MIN:0.01 min:0.01 mean:0.02 max:0.04 LIFE_MAX:0.07
[04/Oct/2017 21:38:20 +0000] 5147 MonitorDaemon-Reporter throttling_logger ERROR    (10 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-58d97c64d44220f47b91dc6ed632f373
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.1-py2.7.egg/cmf/monitor/firehose.py", line 116, in _send
    self._port)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__
    self.conn.connect()
  File "/usr/lib/python2.7/httplib.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 575, in create_connection
    raise err
error: [Errno 111] Connection refused
[04/Oct/2017 21:40:09 +0000] 5147 MainThread heartbeat_tracker INFO     HB stats (seconds): num:40 LIFE_MIN:0.01 min:0.00 mean:0.01 max:0.02 LIFE_MAX:0.07

The host address i am using is 127.0.1.1 with host name as my system name. I have tried it with 127.0.0.1 but it fails at intial phase heart beat not received.

Highlighted
Champion
Posts: 600
Registered: ‎05-16-2016

Re: Cloudera Manager distribution error

whats your firewall or iptable status ?

Explorer
Posts: 12
Registered: ‎09-25-2017

Re: Cloudera Manager distribution error

yes firewall is disabled and iptables are off all 6 of them
Posts: 455
Topics: 1
Kudos: 106
Solutions: 59
Registered: ‎04-22-2014

Re: Cloudera Manager distribution error

@Abhiravilla,

 

The log you show does not indicate any problems that would prevent parcel distribution.

 

The following exception only indicates that the agent could not connect to the Host Monitor (which is expected if it is not running):

[04/Oct/2017 20:57:20 +0000] 5147 MonitorDaemon-Reporter throttling_logger ERROR    (10 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-58d97c64d44220f47b91dc6ed632f373

Traceback (most recent call last):

  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.1-py2.7.egg/cmf/monitor/firehose.py", line 116, in _send

    self._port)

  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__

    self.conn.connect()

  File "/usr/lib/python2.7/httplib.py", line 836, in connect

    self.timeout, self.source_address)

  File "/usr/lib/python2.7/socket.py", line 575, in create_connection

    raise err

error: [Errno 111] Connection refused

 

How do you know there are heartbeat problems?

Also:

 

- what is your server_host value in /etc/cloudera-scm-agent/config.ini

- what is the output of "hostname -f"

 

 

Explorer
Posts: 12
Registered: ‎09-25-2017

Re: Cloudera Manager distribution error

Sorry for late reply. I don't know if there are any heart beat problems. At first I got the heart beat problem but after logging in as root I solved that.
Explorer
Posts: 12
Registered: ‎09-25-2017

Re: Cloudera Manager distribution error

output of hostname -f is hd

Champion
Posts: 600
Registered: ‎05-16-2016

Re: Cloudera Manager distribution error

[ Edited ]

1. check your cloudera-scm-agent status 

2 . check your server_host value  in /etc/cloudera-scm-agent/config.ini , it should point to your cloduera manager's host. 

New Contributor
Posts: 3
Registered: ‎10-29-2017

Re: Cloudera Manager distribution error

Hi

 

I had the same issue. after spending days i managed to solve it by using hostname instead of the ip address. i managed to add another node too with out an issue. 

 

Hope this helps 

 

 

 

Announcements