Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Kafka monitoring tool per broker/topic?

avatar
Expert Contributor

I am wondering whether there is existing tool that could gather incoming message rate across brokers and drill it down to broker/topic/partition level?

1 ACCEPTED SOLUTION

avatar
Super Guru

@Frank Lu

Apache Kafka monitoring still needs a lot of work and there is not one tool that can handle all community requirements.

I have used a combination of Burrow and Ambari Metrics for the most recent version of HDP 2.4.2 +.

Burrow is a monitoring tool for keeping track of consumer lag in Apache Kafka. It is designed to monitor every consumer group that is committing offsets to either Kafka or Zookeeper, and to monitor every topic and partition consumed by those groups. This provides a comprehensive view of consumer status. Burrow also provides several HTTP request endpoints for getting information about the Kafka cluster and consumers, separate from the lag status. This can be very useful for creating applications that assist with managing your Kafka clusters when it is not convenient (or possible) to run a Java Kafka client.

Please check these articles for starters:

https://community.hortonworks.com/articles/28103/monitoring-kafka-with-burrow.html

https://community.hortonworks.com/articles/36725/kafka-monitoring-per-topic-and-per-broker.html

View solution in original post

5 REPLIES 5

avatar
Super Guru

@Frank Lu

Apache Kafka monitoring still needs a lot of work and there is not one tool that can handle all community requirements.

I have used a combination of Burrow and Ambari Metrics for the most recent version of HDP 2.4.2 +.

Burrow is a monitoring tool for keeping track of consumer lag in Apache Kafka. It is designed to monitor every consumer group that is committing offsets to either Kafka or Zookeeper, and to monitor every topic and partition consumed by those groups. This provides a comprehensive view of consumer status. Burrow also provides several HTTP request endpoints for getting information about the Kafka cluster and consumers, separate from the lag status. This can be very useful for creating applications that assist with managing your Kafka clusters when it is not convenient (or possible) to run a Java Kafka client.

Please check these articles for starters:

https://community.hortonworks.com/articles/28103/monitoring-kafka-with-burrow.html

https://community.hortonworks.com/articles/36725/kafka-monitoring-per-topic-and-per-broker.html

avatar
Expert Contributor

@Constantin Stanca Thanks for the answer.

I am looking at confluent control center (http://www.confluent.io/product/control-center), it looks like have the capability that requested. However, it is not open sourced, and I have not tried to confirm.

I was told Ganglia may be able to check on node based, but haven't checked either. Anyone can confirm?

avatar
Super Guru

@Frank Lu

Confluent is a different platform. That would mean the customer will migrate from using Kafka from HDP to Confluent. Their version of Kafka is their version and not necessary the same version that all open source community is using.

avatar
Expert Contributor

@Constantin Stanca Agree. That's what I meant they are not open sourced. However, it is nice we can have similar capability in the open source field as they have.

avatar
New Contributor

Though it is a delayed response, you can try Kafka-Manager from yahoo team - https://github.com/yahoo/kafka-manager