Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

[#1]

Hi,

Would it be possible to see the offsets committed to zookeeper of a kafka topic when we consumer it using "storm-kafka" integration & is it possbile to edit the offset committed to zookeeper to re-consume the topic messages again?

And can someone help me identify,

Which class of "storm-kafka" does actually write/commits the offset to zookeeper ? &

which class actually fetches the last committed offset when we re-run a storm topology with same client id ?

Thanks a lot.

[#2]

Hi @Raja A Yes, it should be possible. As far as editing is concerned, it is advisable to not do it while topology is running. There is also an option to tell storm-kafka to not use zookeeper offsets. Which version of hdp are you wanting to look at the classes handling the writing and fetching offsets from zookeeper?

[#3]

Thanks for the response @pshah i am using the storm version, 0.9.3.2.2.4.0-2633

can you pls help me with what classes in this version are doing this fetching offsets/editing them into zookeeper.

Thanks very much.

My whole intension is to see the committed offsets and edit them as necessary.

Also, in which version of Storm i can say to storm to not use the zookeeper for offsets ?

[#4 · Accepted solution]

@Raja A https://github.com/hortonworks/storm/blob/HDP-2.2.4.0-tag/external/storm-kafka/src/jvm/storm/kafka/P... is the file where its read and written. The zk path where it writes depends on your SpoutConfig. It is the class at https://github.com/hortonworks/storm/blob/HDP-2.2.4.0-tag/external/storm-kafka/src/jvm/storm/kafka/S...

You can check out the method committedPath at https://github.com/hortonworks/storm/blob/HDP-2.2.4.0-tag/external/storm-kafka/src/jvm/storm/kafka/P...

[#5]

Thanks very much @pshah, i will check those classes!! But, unfortunately, that links you shared navigating me to "page not found"!

But i can still check the classes by decompiling my storm version jars....!!

[#6]

with the help of the classes you mentioned i am able to check the committed kafka offsets!!

I am trying to write an application now to be able to edit the stored offsets!!

thanks a lot for your help!! @pshah

[#7]

@Raja A Are you logged in to github ?

[#8]

I have a github account, but not really active on itt!! please let me know what information you are looking for ? @pshah

[#9]

@Raja A To be able to look at those links you need to be logged in to your github account and have access to those repositories. Can you login and check?

[#10]

@pshah

okay will check by logging in.

Anyways i decompiled in editor and verified the code, and i am good!

Do we have any sample application that does the editing of stored offsets in zookeer ?