We‘re running a stream-batch processes and we‘re using AMS and Grafana to monitor the dataflow and how many messages are being written and read on Kafka.
Is there a way to adjust how often AMS queries Kafka metrics such that they‘re collected every minute?
Kafka "server.properties" file will have the following properties:
kafka.metrics.reporters = org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter kafka.timeline.metrics.reporter.enabled = true kafka.timeline.metrics.reporter.sendInterval = 5900
Ambari metrics Collector does not pull the metrics from Kafka, Rather the "KafkaTimelineMetricsReporter" Sink which is running inside the Kafka Broker JVM Push (sends) the metrics to the Ambari Metrics Collector. More information on how "KafkaTimelineMetricsReporter" can be found here: https://github.com/apache/ambari/blob/trunk/ambari-metrics/ambari-metrics-kafka-sink/src/main/java/o...
Kafka Broker loads this class Example
# lsof -p `cat /var/run/kafka/kafka.pid` | grep ambari-metrics-kafka-sink-with-common # $JAVA_HOME/bin/jar -tvf /usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink-with-common-220.127.116.11.1.jar | grep 'KafkaTimelineMetricsReporter' 10993 Tue May 29 18:36:38 UTC 2018 org/apache/hadoop/metrics2/sink/kafka/KafkaTimelineMetricsReporter.class
Thanks for the help ... I tried then to decrease the minute aggregation interval for the AMS from 300 second to 60 but still didn’t bring about anything ... what’s a good way to have a 1 minute lag for the data flow monitoring on Kafka?