Missing Hive Metrics after HDP-3 upgrade

[#1]

In HDP-2.6/Ambari-2.6, it was not mandatory enable HS2 metrics explicitly. Thus, all metrics would be emitted without defining any configs explicitly.

In HDP-3/Ambari-2.7, we will see similar erros in AMS Collector Log:

Error : 2019-06-10 02:42:59,215 INFO timeline timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20

Debug Error shows this :
2019-06-14 20:35:29,538 DEBUG main timeline.HadoopTimelineMetricsSink: Trying to find live collector host from : exp5.lab.com,exp4.lab.com 
2019-06-14 20:35:29,538 DEBUG main timeline.HadoopTimelineMetricsSink: Requesting live collector nodes : http://exp5.lab.com,exp4.lab.com:6188/ws/v1/timeline/metrics/livenodes 
2019-06-14 20:35:29,557 DEBUG main timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://exp5.lab.com,exp4.lab.com:6188/ws/v1/timeline/metrics/livenodes 
2019-06-14 20:35:29,557 DEBUG main timeline.HadoopTimelineMetricsSink: java.net.UnknownHostException: exp5.lab.com,exp4.lab.com 
2019-06-14 20:35:29,558 DEBUG main timeline.HadoopTimelineMetricsSink: Collector exp5.lab.com,exp4.lab.com is not longer live. Removing it from list of know live collector hosts : [] 2019-06-14 20:35:29,558 DEBUG main timeline.HadoopTimelineMetricsSink: No live collectors from configuration.

You need to ensure the following properties exist. If not, first add them in the respective custom section via Ambari >Hive> Configs. 

Next, if you are using Ambari Metrics with more than one collector, then you need to make one more change due a BUG, which will likely be fixed after Ambari-2.7.4.

Add 

*.sink.timeline.zookeeper.quorum=<ZK_QUORUM_ADDRESS>

Example:
*.sink.timeline.zookeeper.quorum=zk_host1:2181,zk_host2:2181,zk_host3:2181

to all the 4 files under 

/var/lib/ambari-server/resources/stacks/HDP/3.0/services/HIVE/package/templates/

located on Ambari Server host.

Restart Ambari Server & Hive for changes to take effect.

Now the metrics will be emitted and you should be able to see data on your Grafana Dashboard.