Missing HBase metrics after HDP-3 Upgrade

[#1]

When using Ambari Metrics in Distributed Mode,after HDP-3/Ambari-2.7 upgrade, HBase metrics are not emitted due to an issue which will likely be fixed after Ambari-2.7.4.

We will see similar kind of messages in AMS Collector Log.

Error : 2019-06-10 02:42:59,215 INFO timeline timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20

Debug Error shows this :
2019-06-14 20:35:29,538 DEBUG main timeline.HadoopTimelineMetricsSink: Trying to find live collector host from : exp5.lab.com,exp4.lab.com 
2019-06-14 20:35:29,538 DEBUG main timeline.HadoopTimelineMetricsSink: Requesting live collector nodes : http://exp5.lab.com,exp4.lab.com:6188/ws/v1/timeline/metrics/livenodes 
2019-06-14 20:35:29,557 DEBUG main timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://exp5.lab.com,exp4.lab.com:6188/ws/v1/timeline/metrics/livenodes 
2019-06-14 20:35:29,557 DEBUG main timeline.HadoopTimelineMetricsSink: java.net.UnknownHostException: exp5.lab.com,exp4.lab.com 
2019-06-14 20:35:29,558 DEBUG main timeline.HadoopTimelineMetricsSink: Collector exp5.lab.com,exp4.lab.com is not longer live. Removing it from list of know live collector hosts : [] 2019-06-14 20:35:29,558 DEBUG main timeline.HadoopTimelineMetricsSink: No live collectors from configuration.

Basically, it's incorrectly parsing hostnames when there are more than one Metrics collector.

In the meantime, there is a very easy workaround.

Add 

*.sink.timeline.zookeeper.quorum=<ZK_QUORUM_ADDRESS>

Example:
*.sink.timeline.zookeeper.quorum=zk_host1:2181,zk_host2:2181,zk_host3:2181

to the following files on Ambari Server host:

/var/lib/ambari-server/resources/stacks/HDP/3.0/services/HBASE/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2

/var/lib/ambari-server/resources/stacks/HDP/3.0/services/HBASE/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2

Restart Ambari Server for changes to take effect and soon you will be able to see metrics on the Grafana HBase Dashboards.

[#2]

@dineshc 

Please specify where the mentioned workaround property needs to be added.

Ams-site.xml or Ams-Hbase-site.xml?