Trying to monitor My hdp 2.6 kafka cluster with Ambari, ( also looking to the grafana dashboards)
i do not find any feature allowing to monitor the cluster at a fine grained level
( i mean managing end2end delivery , reacting with alerts, or perfs management --> all from a GUI
as the Control Center (from Confluence) allows.
so i tried iinstall/set up the CCC on my HDP and i experiment some
start up issues
despite the changes in properties ( kafka via ambari) and control center
i get the rocket ship graphic in the web interface ( see ) https://docs.confluent.io/current/control-center/docs/installation/troubleshooting.html#system-healt...
Before going further , i was wondering about compatiblity issues bettween CCC and HDP kafka
My HW/SW context
Confluent Control Center ( confluent 3.3.0) +
HDP 2.6 ( kafka 0.10.1.2.6.3.0) on a linux VM ubuntu 16.04.2
Confluent Control Center requires an enterprise license in the long-term, which if you are using, you probably should install and maintain your own Kafka installation outside of HDP; maybe sharing a Zookeeper.
In any case, Confluent 3.3.0 is for Kafka version 0.11.0.0, which means your brokers are old. Also, the Kafka topic that control center listens to requires having Confluent's packages on your Kafka classpath. Therefore, while maybe possible, it needs to be distributed on all Kafka brokers and Connect workers, but Ambari isn't going to do that just by changing some configurations.
If you want some type of alerting and performance monitoring, simply exposing JMX metrics can provide lots of useful information like In-Sync Replicas, Partition Count per broker, Total Bytes In/Out, message throughput. For example, see https://www.robustperception.io/monitoring-kafka-with-prometheus/
i was not aware about the constraint regarding 0.11 kafka version.
you are right, an approach with JMX metrics exploited through external Apps can be nice for
the long term ..