Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari Collector start failed on Ambari 2.0.0

Ambari Collector start failed on Ambari 2.0.0

New Contributor

Dear All,

I had just builded Hortonworks systems with Ambari 2.0.0 and HDP 2.2.4. It happened some error, that Ambari Collector not started

3368-capture.png

Any information about ambari:

  • hbase.rootdir: /var/lib/ambari-metrics-collector/hbase-tm
  • Metrics Service operation mode: embedded
  • hbase.cluster.distributed: false
  • hbase.zookeeper.property.clientPort: 61181
  • hbase.zookeeper.quorum: {{zookeeper_quorum_hosts}}

And I have log below:

* File /var/log/ambari-metrics-collector/hbase-ams-master-mt03.abc.vn.log

[root@mt03 hbase-tmp]# tail -300f /var/log/ambari-metrics-collector/hbase-ams-master-mt03.abc.vn.log 
2016-04-12 13:52:35,110 INFO  [timeline] httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2016-04-12 13:52:35,110 INFO  [timeline] httpclient.HttpMethodDirector: Retrying request
2016-04-12 13:52:35,110 INFO  [timeline] httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2016-04-12 13:52:35,110 INFO  [timeline] httpclient.HttpMethodDirector: Retrying request
2016-04-12 13:52:35,110 INFO  [timeline] httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2016-04-12 13:52:35,461 WARN  [MASTER_META_SERVER_OPERATIONS-mt03:54301-0] master.AssignmentManager: Can't move 1588230740, there is no destination server available.
2016-04-12 13:52:35,461 WARN  [MASTER_META_SERVER_OPERATIONS-mt03:54301-0] master.AssignmentManager: Unable to determine a plan to assign {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
2016-04-12 13:52:36,461 WARN  [MASTER_META_SERVER_OPERATIONS-mt03:54301-0] master.AssignmentManager: Can't move 1588230740, there is no destination server available.
2016-04-12 13:52:36,461 WARN  [MASTER_META_SERVER_OPERATIONS-mt03:54301-0] master.AssignmentManager: Unable to determine a plan to assign {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
2016-04-12 13:52:37,462 WARN  [MASTER_META_SERVER_OPERATIONS-mt03:54301-0] master.AssignmentManager: Can't move 1588230740, there is no destination server available.
2016-04-12 13:52:37,462 WARN  [MASTER_META_SERVER_OPERATIONS-mt03:54301-0] master.AssignmentManager: Unable to determine a plan to assign {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
2016-04-12 13:52:38,462 WARN  [MASTER_META_SERVER_OPERATIONS-mt03:54301-0] master.AssignmentManager: Can't move 1588230740, there is no destination server available.
2016-04-12 13:52:38,462 WARN  [MASTER_META_SERVER_OPERATIONS-mt03:54301-0] master.AssignmentManager: Unable to determine a plan to assign {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}

* File /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out

2016-04-12 13:30:30,437 [INFO] emitter.py:59 - Retrying emit after 5 seconds.
2016-04-12 13:30:39,700 [INFO] emitter.py:88 - server: http://mt03.abc.vn:6188/ws/v1/timeline/metrics
2016-04-12 13:30:39,773 [WARNING] emitter.py:54 - Unable to emit events. <urlopen error [Errno 111] Connection refused>
2016-04-12 13:30:44,774 [INFO] emitter.py:59 - Retrying emit after 5 seconds.
2016-04-12 13:30:54,148 [INFO] emitter.py:88 - server: http://mt03.abc.vn:6188/ws/v1/timeline/metrics
2016-04-12 13:30:54,232 [WARNING] emitter.py:54 - Unable to emit events. <urlopen error [Errno 111] Connection refused>
2016-04-12 13:30:59,233 [INFO] emitter.py:59 - Retrying emit after 5 seconds.
2016-04-12 13:31:08,238 [INFO] emitter.py:88 - server: http://mt03.abc.vn:6188/ws/v1/timeline/metrics
2016-04-12 13:31:08,323 [WARNING] emitter.py:54 - Unable to emit events. <urlopen error [Errno 111] Connection refused>

Please, You help me!

Thanks,

3 REPLIES 3

Re: Ambari Collector start failed on Ambari 2.0.0

@tunglq it, hbase.rootdir requires a schema, add "file://"

hbase.rootdir: file:///var/lib/ambari-metrics-collector/hbase-tm

Re: Ambari Collector start failed on Ambari 2.0.0

New Contributor

I still set default value of hbase.rootdir and execute below recomment and I resoved problem.

Thanks Minovic!

Re: Ambari Collector start failed on Ambari 2.0.0

New Contributor

Thanks Emil with your answer!

I increased ambari metrics heapsize to 1024m, kill all ams's process and remove all folder into /var/lib/ambari-metrics-collector/hbase-tmp, that resoved my problem.

Thanks again,