Community Articles
Find and share helpful community-sourced technical articles.

When using Ambari Metrics in Distributed Mode,after HDP-3/Ambari-2.7 upgrade, HBase metrics are not emitted due to an issue which will likely be fixed after Ambari-2.7.4.

 

We will see similar kind of messages in AMS Collector Log.

Error : 2019-06-10 02:42:59,215 INFO timeline timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20
Debug Error shows this : 2019-06-14 20:35:29,538 DEBUG main timeline.HadoopTimelineMetricsSink: Trying to find live collector host from : exp5.lab.com,exp4.lab.com
2019-06-14 20:35:29,538 DEBUG main timeline.HadoopTimelineMetricsSink: Requesting live collector nodes : http://exp5.lab.com,exp4.lab.com:6188/ws/v1/timeline/metrics/livenodes
2019-06-14 20:35:29,557 DEBUG main timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://exp5.lab.com,exp4.lab.com:6188/ws/v1/timeline/metrics/livenodes
2019-06-14 20:35:29,557 DEBUG main timeline.HadoopTimelineMetricsSink: java.net.UnknownHostException: exp5.lab.com,exp4.lab.com
2019-06-14 20:35:29,558 DEBUG main timeline.HadoopTimelineMetricsSink: Collector exp5.lab.com,exp4.lab.com is not longer live. Removing it from list of know live collector hosts : [] 2019-06-14 20:35:29,558 DEBUG main timeline.HadoopTimelineMetricsSink: No live collectors from configuration.

Basically, it's incorrectly parsing hostnames when there are more than one Metrics collector.

 

In the meantime, there is a very easy workaround.

 

Add 

*.sink.timeline.zookeeper.quorum=<ZK_QUORUM_ADDRESS>

Example:
*.sink.timeline.zookeeper.quorum=zk_host1:2181,zk_host2:2181,zk_host3:2181

 

to the following files on Ambari Server host:

/var/lib/ambari-server/resources/stacks/HDP/3.0/services/HBASE/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2

/var/lib/ambari-server/resources/stacks/HDP/3.0/services/HBASE/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2

Restart Ambari Server for changes to take effect and soon you will be able to see metrics on the Grafana HBase Dashboards.

630 Views
Comments
Cloudera Employee

@dineshc 

Please specify where the mentioned workaround property needs to be added.

Ams-site.xml or Ams-Hbase-site.xml?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.
Version history
Last update:
‎08-22-2019 02:21 PM
Updated by:
Contributors
Top Kudoed Authors
; ;