question Additional Ambari metrics - Kafka in Support Questions

Additional Ambari metrics - Kafka

james_jones — Mon, 16 May 2016 22:30:43 GMT

I'm looking for a way to integrate some of the Kafka JMX metrics into Ambari monitoring and alerting if possible. The installation I will be working with is HDP 2.3.4, but at the moment I'm looking at the 2.4 sandbox. In the sandbox, I see these two files which are very similar:

/var/lib/ambari-server/resources/stacks/HDP/2.3/services/KAFKA/metrics.json

/var/lib/ambari-server/resources/common-services/KAFKA/0.8.1.2.2/metrics.json

They appear to be for collecting JMX metrics, but I'm not sure how to take advantage of this. Is it possible?

Thanks

Re: Additional Ambari metrics - Kafka

james_jones — Tue, 17 May 2016 01:41:00 GMT

These files have stuff like this in them, so how do I get this info in Ambari?

{
  "KAFKA_BROKER": {
    "Component": [
      {
        "type": "ganglia",
        "metrics": {
          "default": {
            "metrics/jvm/uptime": {
              "metric": "jvm.uptime",
              "pointInTime": true,
              "temporal": true
            },

/* -- SNIP -- */
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesInPerSec/count": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesInPerSec.count",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/1MinuteRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.1MinuteRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/5MinuteRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.5MinuteRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/15MinuteRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/meanRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.meanRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/count": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.count",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/controller/KafkaController/ActiveControllerCount": {
              "metric": "kafka.controller.KafkaController.ActiveControllerCount",
              "pointInTime": true,
              "temporal": true
            },

Re: Additional Ambari metrics - Kafka

sidwagle — Tue, 17 May 2016 02:04:49 GMT

Ambari 2.2.2 comes with built in Grafana integration so you can build a dashboard of you choice. All metrics are self discovered and no extra steps are required.

Adding Grafana:

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_upgrading_Ambari/content/_adding_grafana_to_ambari_metrics.html

Grafana docs:

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Ambari_Users_Guide/content/_using_grafana.html

The Ambari widgets provide a service level aggregated view of the service metrics. You would need to update the metrics.json file from "/var/lib/ambari-server/resources/stacks/HDP/2.3/services/KAFKA/metrics.json", this overrides the common-services definition for HDP versions 2.3+ And then add widget from Ambari.

Docs for that:

https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Service+Dashboard

Re: Additional Ambari metrics - Kafka

james_jones — Tue, 17 May 2016 09:47:19 GMT

Thanks for the info @swagle.

We were able to convince the customer to upgrade to Ambari 2.2.2. They were thrilled with the fact that you can add widgets directly in the UI, and the fact that Grafana is available is icing on the cake. Incidentally, here is the JIRA for adding this for Kafka and Storm: https://issues.apache.org/jira/browse/AMBARI-14941

Re: Additional Ambari metrics - Kafka

sidwagle — Tue, 17 May 2016 09:57:13 GMT

KAFKA and STORM default dashboards are in the works and targeted for 2.4.0

However you can start visualizing these metrics with custom dashboard from 2.2.2 itself.

Re: Additional Ambari metrics - Kafka

ArtiW — Wed, 12 Oct 2016 00:31:17 GMT

Hey @swagle..

Do you have any link for setting up Kafka custom dashboard in Grafana for Ambari 2.2.2?

Re: Additional Ambari metrics - Kafka

sidwagle — Wed, 12 Oct 2016 00:53:17 GMT

Actually, Grafana allow dashboard import. You might be able to just get the dashboard definition from Ambari source and import into you existing Grafana.

Here is link to trunk dashboard locations:

https://github.com/apache/ambari/tree/trunk/ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/files/grafana-dashboards/HDP

Import is pretty straightforward:

http://docs.grafana.org/reference/export_import/