Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Additional Ambari metrics - Kafka

avatar
Super Collaborator

I'm looking for a way to integrate some of the Kafka JMX metrics into Ambari monitoring and alerting if possible. The installation I will be working with is HDP 2.3.4, but at the moment I'm looking at the 2.4 sandbox. In the sandbox, I see these two files which are very similar:

/var/lib/ambari-server/resources/stacks/HDP/2.3/services/KAFKA/metrics.json

/var/lib/ambari-server/resources/common-services/KAFKA/0.8.1.2.2/metrics.json

They appear to be for collecting JMX metrics, but I'm not sure how to take advantage of this. Is it possible?

Thanks

1 ACCEPTED SOLUTION

avatar
Super Collaborator

Ambari 2.2.2 comes with built in Grafana integration so you can build a dashboard of you choice. All metrics are self discovered and no extra steps are required.

Adding Grafana:

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_upgrading_Ambari/content/_adding_grafana...

Grafana docs:

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Ambari_Users_Guide/content/_using_grafana...

The Ambari widgets provide a service level aggregated view of the service metrics. You would need to update the metrics.json file from "/var/lib/ambari-server/resources/stacks/HDP/2.3/services/KAFKA/metrics.json", this overrides the common-services definition for HDP versions 2.3+ And then add widget from Ambari.

Docs for that:

https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Service+Dashboard

View solution in original post

6 REPLIES 6

avatar
Super Collaborator

These files have stuff like this in them, so how do I get this info in Ambari?

{
  "KAFKA_BROKER": {
    "Component": [
      {
        "type": "ganglia",
        "metrics": {
          "default": {
            "metrics/jvm/uptime": {
              "metric": "jvm.uptime",
              "pointInTime": true,
              "temporal": true
            },

/* -- SNIP -- */
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesInPerSec/count": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesInPerSec.count",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/1MinuteRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.1MinuteRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/5MinuteRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.5MinuteRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/15MinuteRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/meanRate": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.meanRate",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/count": {
              "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.count",
              "pointInTime": true,
              "temporal": true
            },
            "metrics/kafka/controller/KafkaController/ActiveControllerCount": {
              "metric": "kafka.controller.KafkaController.ActiveControllerCount",
              "pointInTime": true,
              "temporal": true
            },

avatar
Super Collaborator

Ambari 2.2.2 comes with built in Grafana integration so you can build a dashboard of you choice. All metrics are self discovered and no extra steps are required.

Adding Grafana:

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_upgrading_Ambari/content/_adding_grafana...

Grafana docs:

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Ambari_Users_Guide/content/_using_grafana...

The Ambari widgets provide a service level aggregated view of the service metrics. You would need to update the metrics.json file from "/var/lib/ambari-server/resources/stacks/HDP/2.3/services/KAFKA/metrics.json", this overrides the common-services definition for HDP versions 2.3+ And then add widget from Ambari.

Docs for that:

https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Service+Dashboard

avatar
Super Collaborator

Thanks for the info @swagle.

We were able to convince the customer to upgrade to Ambari 2.2.2. They were thrilled with the fact that you can add widgets directly in the UI, and the fact that Grafana is available is icing on the cake. Incidentally, here is the JIRA for adding this for Kafka and Storm: https://issues.apache.org/jira/browse/AMBARI-14941

avatar
Super Collaborator

KAFKA and STORM default dashboards are in the works and targeted for 2.4.0

However you can start visualizing these metrics with custom dashboard from 2.2.2 itself.

avatar
Rising Star

Hey @swagle..

Do you have any link for setting up Kafka custom dashboard in Grafana for Ambari 2.2.2?

avatar
Super Collaborator

Actually, Grafana allow dashboard import. You might be able to just get the dashboard definition from Ambari source and import into you existing Grafana.

Here is link to trunk dashboard locations:

https://github.com/apache/ambari/tree/trunk/ambari-server/src/main/resources/common-services/AMBARI_...

Import is pretty straightforward:

http://docs.grafana.org/reference/export_import/