Support Questions
Find answers, ask questions, and share your expertise

How to change AMS Kafka metrics update interval

How to change AMS Kafka metrics update interval

New Contributor

Hi,

We‘re running a stream-batch processes and we‘re using AMS and Grafana to monitor the dataflow and how many messages are being written and read on Kafka.

Is there a way to adjust how often AMS queries Kafka metrics such that they‘re collected every minute?

3 REPLIES 3

Re: How to change AMS Kafka metrics update interval

Super Mentor
@Mohamed Maamoun

Kafka "server.properties" file will have the following properties:

kafka.metrics.reporters = org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter
kafka.timeline.metrics.reporter.enabled = true   
kafka.timeline.metrics.reporter.sendInterval = 5900


Ambari metrics Collector does not pull the metrics from Kafka, Rather the "KafkaTimelineMetricsReporter" Sink which is running inside the Kafka Broker JVM Push (sends) the metrics to the Ambari Metrics Collector. More information on how "KafkaTimelineMetricsReporter" can be found here: https://github.com/apache/ambari/blob/trunk/ambari-metrics/ambari-metrics-kafka-sink/src/main/java/o...

Kafka Broker loads this class Example

# lsof -p `cat /var/run/kafka/kafka.pid` | grep ambari-metrics-kafka-sink-with-common
# $JAVA_HOME/bin/jar -tvf /usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink-with-common-2.6.2.2.1.jar  | grep 'KafkaTimelineMetricsReporter'
 10993 Tue May 29 18:36:38 UTC 2018 org/apache/hadoop/metrics2/sink/kafka/KafkaTimelineMetricsReporter.class

.

Re: How to change AMS Kafka metrics update interval

New Contributor

@jay kumar

Thanks for the help ... I tried then to decrease the minute aggregation interval for the AMS from 300 second to 60 but still didn’t bring about anything ... what’s a good way to have a 1 minute lag for the data flow monitoring on Kafka?

Re: How to change AMS Kafka metrics update interval

Very nice and interesting
https://www.welookups.com