How to limit kafka metrics so as to not crash the ambari metrics collector

[#1]

Issue:

In a heavily utilized kafka cluster AMS will keep on crashing with error

"ERROR org.apache.hadoop.hbase.client.AsyncProcess: Cannot get replica 0 location for {"totalColumns":5,"row":"kafka.server.FetcherLagMetrics."

Solution:

1. Run the following command to gather the amount of metrics being collected:

curl http://<Ambari-metrics-collector-host>:6188/ws/v1/timeline/metrics/metadata

2. From Ambari UI -> Kafka -> Configs -> Fliter

search for: "external.kafka.metrics.exclude.prefix"

3. Add the following at the end:

kafka.log.Log

4. Restart Kafka.

This will exclude additional metrics from getting captured and will increase the stability of the AMS.

[#2]

We have two filter options external.kafka.metrics.exclude.prefix and external.kafka.metrics.include.prefix.

https://github.com/hortonworks/ambari/blob/AMBARI-2.7.0-maint/ambari-metrics/ambari-metrics-kafka-si...