Additional Ambari metrics - Kafka

[#1]

I'm looking for a way to integrate some of the Kafka JMX metrics into Ambari monitoring and alerting if possible. The installation I will be working with is HDP 2.3.4, but at the moment I'm looking at the 2.4 sandbox. In the sandbox, I see these two files which are very similar:

/var/lib/ambari-server/resources/stacks/HDP/2.3/services/KAFKA/metrics.json

/var/lib/ambari-server/resources/common-services/KAFKA/0.8.1.2.2/metrics.json

They appear to be for collecting JMX metrics, but I'm not sure how to take advantage of this. Is it possible?

Thanks

[#2]

These files have stuff like this in them, so how do I get this info in Ambari?

{
  "KAFKA_BROKER": {
    "Component": [
      {
        "type": "ganglia",
        "metrics": {
          "default": {
            "metrics/jvm/uptime": {
              "metric": "jvm.uptime",
              "pointInTime": true,
              "temporal": true
            },

/* -- SNIP -- */
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesInPerSec/count": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesInPerSec.count",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/1MinuteRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.1MinuteRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/5MinuteRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.5MinuteRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/15MinuteRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/meanRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.meanRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/count": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.count",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/controller/KafkaController/ActiveControllerCount": {
              "metric": "kafka.controller.KafkaController.ActiveControllerCount",
              "pointInTime": true,
              "temporal": true
            },

[#3 · Accepted solution]

Ambari 2.2.2 comes with built in Grafana integration so you can build a dashboard of you choice. All metrics are self discovered and no extra steps are required.

Adding Grafana:

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_upgrading_Ambari/content/_adding_grafana...

Grafana docs:

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Ambari_Users_Guide/content/_using_grafana...

The Ambari widgets provide a service level aggregated view of the service metrics. You would need to update the metrics.json file from "/var/lib/ambari-server/resources/stacks/HDP/2.3/services/KAFKA/metrics.json", this overrides the common-services definition for HDP versions 2.3+ And then add widget from Ambari.

Docs for that:

https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Service+Dashboard

[#4]

Thanks for the info @swagle.

We were able to convince the customer to upgrade to Ambari 2.2.2. They were thrilled with the fact that you can add widgets directly in the UI, and the fact that Grafana is available is icing on the cake. Incidentally, here is the JIRA for adding this for Kafka and Storm: https://issues.apache.org/jira/browse/AMBARI-14941

[#5]

KAFKA and STORM default dashboards are in the works and targeted for 2.4.0

However you can start visualizing these metrics with custom dashboard from 2.2.2 itself.

[#6]

Hey @swagle..

Do you have any link for setting up Kafka custom dashboard in Grafana for Ambari 2.2.2?

[#7]

Actually, Grafana allow dashboard import. You might be able to just get the dashboard definition from Ambari source and import into you existing Grafana.

Here is link to trunk dashboard locations:

https://github.com/apache/ambari/tree/trunk/ambari-server/src/main/resources/common-services/AMBARI_...

Import is pretty straightforward:

http://docs.grafana.org/reference/export_import/