- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Kafka monitoring tool per broker/topic?
- Labels:
-
Apache Kafka
Created 08-10-2016 01:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am wondering whether there is existing tool that could gather incoming message rate across brokers and drill it down to broker/topic/partition level?
Created 08-10-2016 01:23 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Apache Kafka monitoring still needs a lot of work and there is not one tool that can handle all community requirements.
I have used a combination of Burrow and Ambari Metrics for the most recent version of HDP 2.4.2 +.
Burrow is a monitoring tool for keeping track of consumer lag in Apache Kafka. It is designed to monitor every consumer group that is committing offsets to either Kafka or Zookeeper, and to monitor every topic and partition consumed by those groups. This provides a comprehensive view of consumer status. Burrow also provides several HTTP request endpoints for getting information about the Kafka cluster and consumers, separate from the lag status. This can be very useful for creating applications that assist with managing your Kafka clusters when it is not convenient (or possible) to run a Java Kafka client.
Please check these articles for starters:
https://community.hortonworks.com/articles/28103/monitoring-kafka-with-burrow.html
https://community.hortonworks.com/articles/36725/kafka-monitoring-per-topic-and-per-broker.html
Created 08-10-2016 01:23 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Apache Kafka monitoring still needs a lot of work and there is not one tool that can handle all community requirements.
I have used a combination of Burrow and Ambari Metrics for the most recent version of HDP 2.4.2 +.
Burrow is a monitoring tool for keeping track of consumer lag in Apache Kafka. It is designed to monitor every consumer group that is committing offsets to either Kafka or Zookeeper, and to monitor every topic and partition consumed by those groups. This provides a comprehensive view of consumer status. Burrow also provides several HTTP request endpoints for getting information about the Kafka cluster and consumers, separate from the lag status. This can be very useful for creating applications that assist with managing your Kafka clusters when it is not convenient (or possible) to run a Java Kafka client.
Please check these articles for starters:
https://community.hortonworks.com/articles/28103/monitoring-kafka-with-burrow.html
https://community.hortonworks.com/articles/36725/kafka-monitoring-per-topic-and-per-broker.html
Created 08-10-2016 01:45 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Constantin Stanca Thanks for the answer.
I am looking at confluent control center (http://www.confluent.io/product/control-center), it looks like have the capability that requested. However, it is not open sourced, and I have not tried to confirm.
I was told Ganglia may be able to check on node based, but haven't checked either. Anyone can confirm?
Created 08-10-2016 03:34 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Confluent is a different platform. That would mean the customer will migrate from using Kafka from HDP to Confluent. Their version of Kafka is their version and not necessary the same version that all open source community is using.
Created 08-10-2016 03:42 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Constantin Stanca Agree. That's what I meant they are not open sourced. However, it is nice we can have similar capability in the open source field as they have.
Created 02-21-2019 10:12 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Though it is a delayed response, you can try Kafka-Manager from yahoo team - https://github.com/yahoo/kafka-manager