- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!
- Labels:
-
Apache Kafka
-
Apache Storm
Created ‎05-11-2016 10:01 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Would it be possible to see the offsets committed to zookeeper of a kafka topic when we consumer it using "storm-kafka" integration & is it possbile to edit the offset committed to zookeeper to re-consume the topic messages again?
And can someone help me identify,
Which class of "storm-kafka" does actually write/commits the offset to zookeeper ? &
which class actually fetches the last committed offset when we re-run a storm topology with same client id ?
Thanks a lot.
Created ‎05-12-2016 05:40 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Raja A https://github.com/hortonworks/storm/blob/HDP-2.2.4.0-tag/external/storm-kafka/src/jvm/storm/kafka/P... is the file where its read and written. The zk path where it writes depends on your SpoutConfig. It is the class at https://github.com/hortonworks/storm/blob/HDP-2.2.4.0-tag/external/storm-kafka/src/jvm/storm/kafka/S...
You can check out the method committedPath at https://github.com/hortonworks/storm/blob/HDP-2.2.4.0-tag/external/storm-kafka/src/jvm/storm/kafka/P...
Created ‎05-16-2016 08:17 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Raja A Not that i know of. You can edit the offsets using command line zookeeper client. In a test or dev setup this should be fine. I would not advise it in a production environment unless you know for sure that updating the offsets will not cause storm application to skip data for processing.

- « Previous
-
- 1
- 2
- Next »