Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

avatar
Expert Contributor

Hi,

Would it be possible to see the offsets committed to zookeeper of a kafka topic when we consumer it using "storm-kafka" integration & is it possbile to edit the offset committed to zookeeper to re-consume the topic messages again?

And can someone help me identify,

Which class of "storm-kafka" does actually write/commits the offset to zookeeper ? &

which class actually fetches the last committed offset when we re-run a storm topology with same client id ?

Thanks a lot.

1 ACCEPTED SOLUTION
10 REPLIES 10

avatar
Rising Star

Hi @Raja A Yes, it should be possible. As far as editing is concerned, it is advisable to not do it while topology is running. There is also an option to tell storm-kafka to not use zookeeper offsets. Which version of hdp are you wanting to look at the classes handling the writing and fetching offsets from zookeeper?

avatar
Expert Contributor

Thanks for the response @pshah i am using the storm version, 0.9.3.2.2.4.0-2633

can you pls help me with what classes in this version are doing this fetching offsets/editing them into zookeeper.

Thanks very much.

My whole intension is to see the committed offsets and edit them as necessary.

Also, in which version of Storm i can say to storm to not use the zookeeper for offsets ?

avatar
Rising Star

avatar
Expert Contributor

Thanks very much @pshah, i will check those classes!! But, unfortunately, that links you shared navigating me to "page not found"!

But i can still check the classes by decompiling my storm version jars....!!

avatar
Expert Contributor

with the help of the classes you mentioned i am able to check the committed kafka offsets!!

I am trying to write an application now to be able to edit the stored offsets!!

thanks a lot for your help!! @pshah

avatar
Rising Star

@Raja A Are you logged in to github ?

avatar
Expert Contributor

I have a github account, but not really active on itt!! please let me know what information you are looking for ? @pshah

avatar
Rising Star

@Raja A To be able to look at those links you need to be logged in to your github account and have access to those repositories. Can you login and check?

avatar
Expert Contributor

@pshah

okay will check by logging in.

Anyways i decompiled in editor and verified the code, and i am good!

Do we have any sample application that does the editing of stored offsets in zookeer ?