Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Guru

If you’re using Apache Kafka, you know it persists all the messages on disk as a distributed commit log. You might sometimes want to take advantage of that and reprocess some of the messages. Assuming that you want to reprocess all the messages currently stored on your brokers and you set auto.offset.reset to smallest, you can just delete your consumers’ data from Zookeeper. After restarting, your consumers should start from the beginning. But what if you forgot or didn’t want to set auto.offset.reset in you consumers to smallest? Then you can manually set the offsets for each partition for your consumers to the smallest currently available offset.

List all topics

kafka-topics --list --zookeeper localhost:2181

12107-screen-shot-2017-01-20-at-71914-am.png

Get Offsets for the topic

kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic vital_signs --time -1

12108-screen-shot-2017-01-20-at-72457-am.png

Set the offset manually

$ ZOOKEEPER_HOME/zkCli
$ ls /consumers/flasfka/offsets/vital_signs

$ set /consumers/flasfka/offsets/vital_signs/0 11492949

12109-screen-shot-2017-01-20-at-74657-am.png

Verify the new offset

kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic vital_signs --time -1

12110-screen-shot-2017-01-20-at-73457-am.png

62,785 Views
Comments
Super Collaborator

@Vedant Jain Very interesting article. And usefull

I am looking for a way though to do to same for NewConsumers that store their offset in the topic __consumer_offsets.

Do you know how to manage that offset?

New Contributor

That's pretty useful, but only if you're storing the offsets in Zookeeper. Can someone please update this article with instructions for how to reset offsets when they are stored in Kafka?

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 05:08 AM
Updated by:
 
Contributors
Top Kudoed Authors