How can i fund the number of messages published to a kafka topic per hour

[#1]

Hi All , 

How can i fund the number of messages published to a kafka topic per hour ?

When i got to grafana dashboard from ambari , i could see messages in/s ,does this metric give the number of messages at the time frame in kafka topic ?

[#2]

Hi @sarm 

I think there is no metric for that, on the other hand, you can create a simple java consumer and add the following details:

// consumer details here

while (true) {
    try {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));


        for (ConsumerRecord<String, String> record : records) {
            Date date;
            System.out.println(date = new Date(record.timestamp()));
            System.out.printf("Partition = %d\n",record.partition());
            System.out.printf("Offset = %d\n", record.offset());
            System.out.printf("Key    = %s\n", record.key());
            System.out.printf("Value  = %s\n", record.value());
        }
    } catch (Exception e) {
        e.printStackTrace();
    }
}

This should provide the following output:

Wed Nov 24 19:16:27 CLST 2021
Partition = 0
Offset = 439
Key = null
Value = S

 Then you can create some logic to count the number of messages between some specific timing.

I hope that helps.

[#3]

@sarm 

After digging a little bit more, there is a metric exposed by producers called

kafka.producer:type=producer-metrics,client-id=producer-1 > objectName 

record-send-total: This will show the total number of records sent by this producer.

To get more details about the available metrics in Kafka I would suggest checking this Cloudera community article.