question Re: How can i fund the number of messages published to a kafka topic per hour in Support Questions

How can i fund the number of messages published to a kafka topic per hour

sarm — Wed, 10 Nov 2021 15:47:01 GMT

Hi All ,

How can i fund the number of messages published to a kafka topic per hour ?

When i got to grafana dashboard from ambari , i could see messages in/s ,does this metric give the number of messages at the time frame in kafka topic ?

Re: How can i fund the number of messages published to a kafka topic per hour

ManuelCalvo — Wed, 24 Nov 2021 22:22:14 GMT

Hi @sarm

I think there is no metric for that, on the other hand, you can create a simple java consumer and add the following details:

// consumer details here while (true) { try { ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord<String, String> record : records) { Date date; System.out.println(date = new Date(record.timestamp())); System.out.printf("Partition = %d\n",record.partition()); System.out.printf("Offset = %d\n", record.offset()); System.out.printf("Key = %s\n", record.key()); System.out.printf("Value = %s\n", record.value()); } } catch (Exception e) { e.printStackTrace(); } }

This should provide the following output:

Wed Nov 24 19:16:27 CLST 2021 Partition = 0 Offset = 439 Key = null Value = S

Then you can create some logic to count the number of messages between some specific timing.

I hope that helps.

Re: How can i fund the number of messages published to a kafka topic per hour

ManuelCalvo — Tue, 30 Nov 2021 17:48:25 GMT

@sarm

After digging a little bit more, there is a metric exposed by producers called

kafka.producer:type=producer-metrics,client-id=producer-1 > objectName

record-send-total: This will show the total number of records sent by this producer.

To get more details about the available metrics in Kafka I would suggest checking this Cloudera community article.