Kafka Custom alert with ambari 2.2.2

[#1]

Currently ambari alerts for kafka is just to monitor the kafka broker process on a particular port. Is there a way we can monitor and alert any other parameters like under replicated counts or active controller count etc... ? After upgrading to Ambari 2.2.2 we are able to create and add new widgets for these fields. But there is nothing in the alerts section. Has anyone been successful in setting up the custom alerting for these parameters?

[#2]

@Yogeshprabhu

Check this out https://community.hortonworks.com/articles/38149/how-to-create-and-register-custom-ambari-alerts.htm...

[#3 · Accepted solution]

You can definitely create new alerts using the REST APIs, but this is not exposed in the web client yet:

https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/alert-definitions.md#api-summa...

With that said, can you provide the mechanism through which things like "Under Replicate Counts" are exposed through Kafka? If they are JMX or AMS metrics, you can create a METRIC alert to monitor them.

[#4]

Thanks @Jonathan Hurley

I was able to get the "under replicated count" through the AMS Metrics and here is the REST API call.

curl -u  admin:admin -H "X-Requested-by:ambari" -i -k -X GET http://sandbox.hortonworks.com:6188/ws/v1/timeline/metrics?metricNames=kafka.controller.KafkaControl...

[#5]

Well, it seems like AMS metrics aren't consumable by Ambari until version 2.4.0 comes out: https://issues.apache.org/jira/browse/AMBARI-15766

This means that JMX metrics are the only ones which work by default and I don't think Kafka exposes them. You could use a third-party plugin which exposes them, but that's probably not ideal.

So, it seems like to monitor Kafka metrics today you'd need to write a SCRIPT based alert which would check a specific metric from AMS. You could write a single, generic SCRIPT alert which takes the metric to check as a parameter. This way, you'd create several different alert definitions (1 for each metric) and then re-use the same Python script.

If you wanted to expose Kafka metrics via JMX, once of these might help:

https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters

You'd expose JMX via Kafka and then expose that JMX data via something like JMXtrans which alerts could consume.

Or, you could wait for Ambari 2.4.0 which should let you consume AMS metrics natively and just write a simple alert definition for it.