How to change AMS Kafka metrics update interval

[#1]

Hi,

We‘re running a stream-batch processes and we‘re using AMS and Grafana to monitor the dataflow and how many messages are being written and read on Kafka.

Is there a way to adjust how often AMS queries Kafka metrics such that they‘re collected every minute?

[#2]

@Mohamed Maamoun

Kafka "server.properties" file will have the following properties:

kafka.metrics.reporters = org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter
kafka.timeline.metrics.reporter.enabled = true   
kafka.timeline.metrics.reporter.sendInterval = 5900

Ambari metrics Collector does not pull the metrics from Kafka, Rather the "KafkaTimelineMetricsReporter" Sink which is running inside the Kafka Broker JVM Push (sends) the metrics to the Ambari Metrics Collector. More information on how "KafkaTimelineMetricsReporter" can be found here: https://github.com/apache/ambari/blob/trunk/ambari-metrics/ambari-metrics-kafka-sink/src/main/java/o...

Kafka Broker loads this class Example

# lsof -p `cat /var/run/kafka/kafka.pid` | grep ambari-metrics-kafka-sink-with-common
# $JAVA_HOME/bin/jar -tvf /usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink-with-common-2.6.2.2.1.jar  | grep 'KafkaTimelineMetricsReporter'
 10993 Tue May 29 18:36:38 UTC 2018 org/apache/hadoop/metrics2/sink/kafka/KafkaTimelineMetricsReporter.class

.

[#3]

@jay kumar

Thanks for the help ... I tried then to decrease the minute aggregation interval for the AMS from 300 second to 60 but still didn’t bring about anything ... what’s a good way to have a 1 minute lag for the data flow monitoring on Kafka?

[#4]

Very nice and interesting
https://www.welookups.com