Configure/Setup Grafana for Kafka monitoring

[#1]

Hi Folks - I want to setup/configure Grafana for Kafka monitoring and alerting ( For example I want Grafana to send alerts to Admins if there is a consumer-lag, topic unavailability, etc.) .

Want to know if there is any official documentation available for same. Please help !!

Thanks,-Manny

[#2]

@M K

First you need to enable JMX for Kafka by

Edit bin/kafka-run-class.sh

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=your.kafka.broker.hostname -Djava.net.preferIPv4Stack=true"

Update bin/kafka-server-start.sh add the below line and set KAFKA_JMX_OPTS variable

export JMX_PORT=PORT

Once done you can follow https://grafana.com/dashboards/721 for Kafka grafana dashboard metrics.

[#3]

You can't do alerting with the Grafana provided with HDP at the moment. Alerting is only available starting with Grafana v4.0 (see http://docs.grafana.org/alerting/notifications/). You could setup a custom alert in Ambari and use the notification system of Ambari to get alerts.