Support Questions
Find answers, ask questions, and share your expertise

Kafka monitoring tool per broker/topic?

Solved Go to solution
Highlighted

Kafka monitoring tool per broker/topic?

Expert Contributor

I am wondering whether there is existing tool that could gather incoming message rate across brokers and drill it down to broker/topic/partition level?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Kafka monitoring tool per broker/topic?

@Frank Lu

Apache Kafka monitoring still needs a lot of work and there is not one tool that can handle all community requirements.

I have used a combination of Burrow and Ambari Metrics for the most recent version of HDP 2.4.2 +.

Burrow is a monitoring tool for keeping track of consumer lag in Apache Kafka. It is designed to monitor every consumer group that is committing offsets to either Kafka or Zookeeper, and to monitor every topic and partition consumed by those groups. This provides a comprehensive view of consumer status. Burrow also provides several HTTP request endpoints for getting information about the Kafka cluster and consumers, separate from the lag status. This can be very useful for creating applications that assist with managing your Kafka clusters when it is not convenient (or possible) to run a Java Kafka client.

Please check these articles for starters:

https://community.hortonworks.com/articles/28103/monitoring-kafka-with-burrow.html

https://community.hortonworks.com/articles/36725/kafka-monitoring-per-topic-and-per-broker.html

View solution in original post

5 REPLIES 5
Highlighted

Re: Kafka monitoring tool per broker/topic?

@Frank Lu

Apache Kafka monitoring still needs a lot of work and there is not one tool that can handle all community requirements.

I have used a combination of Burrow and Ambari Metrics for the most recent version of HDP 2.4.2 +.

Burrow is a monitoring tool for keeping track of consumer lag in Apache Kafka. It is designed to monitor every consumer group that is committing offsets to either Kafka or Zookeeper, and to monitor every topic and partition consumed by those groups. This provides a comprehensive view of consumer status. Burrow also provides several HTTP request endpoints for getting information about the Kafka cluster and consumers, separate from the lag status. This can be very useful for creating applications that assist with managing your Kafka clusters when it is not convenient (or possible) to run a Java Kafka client.

Please check these articles for starters:

https://community.hortonworks.com/articles/28103/monitoring-kafka-with-burrow.html

https://community.hortonworks.com/articles/36725/kafka-monitoring-per-topic-and-per-broker.html

View solution in original post

Highlighted

Re: Kafka monitoring tool per broker/topic?

Expert Contributor

@Constantin Stanca Thanks for the answer.

I am looking at confluent control center (http://www.confluent.io/product/control-center), it looks like have the capability that requested. However, it is not open sourced, and I have not tried to confirm.

I was told Ganglia may be able to check on node based, but haven't checked either. Anyone can confirm?

Highlighted

Re: Kafka monitoring tool per broker/topic?

@Frank Lu

Confluent is a different platform. That would mean the customer will migrate from using Kafka from HDP to Confluent. Their version of Kafka is their version and not necessary the same version that all open source community is using.

Highlighted

Re: Kafka monitoring tool per broker/topic?

Expert Contributor

@Constantin Stanca Agree. That's what I meant they are not open sourced. However, it is nice we can have similar capability in the open source field as they have.

Highlighted

Re: Kafka monitoring tool per broker/topic?

New Contributor

Though it is a delayed response, you can try Kafka-Manager from yahoo team - https://github.com/yahoo/kafka-manager