Skip to main content
sarm
Silver ⭐️
November 10, 2021
Question

How can i fund the number of messages published to a kafka topic per hour

Hi All , 

 

How can i fund the number of messages published to a kafka topic per hour ?

When i got to grafana dashboard from ambari , i could see messages in/s ,does this metric give the number of messages at the time frame in kafka topic ?

 

2 replies

ManuelCalvo
Gold ⭐️
Gold ⭐️
November 24, 2021

Hi @sarm 

 

I think there is no metric for that, on the other hand, you can create a simple java consumer and add the following details:

 

// consumer details here

while (true) {
try {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));


for (ConsumerRecord<String, String> record : records) {
Date date;
System.out.println(date = new Date(record.timestamp()));
System.out.printf("Partition = %d\n",record.partition());
System.out.printf("Offset = %d\n", record.offset());
System.out.printf("Key = %s\n", record.key());
System.out.printf("Value = %s\n", record.value());
}
} catch (Exception e) {
e.printStackTrace();
}
}

This should provide the following output:

Wed Nov 24 19:16:27 CLST 2021
Partition = 0
Offset = 439
Key = null
Value = S

 Then you can create some logic to count the number of messages between some specific timing.

I hope that helps.

Translate
ManuelCalvo
Gold ⭐️
Gold ⭐️
November 30, 2021

@sarm 

 

After digging a little bit more, there is a metric exposed by producers called

 

kafka.producer:type=producer-metrics,client-id=producer-1 > objectName 

record-send-total: This will show the total number of records sent by this producer.

 

To get more details about the available metrics in Kafka I would suggest checking this Cloudera community article.

 

Translate