question Re: Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!! in Archives of Support Questions (Read Only)

Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

raja_aravapalli — Thu, 12 May 2016 05:01:52 GMT

Hi,

Would it be possible to see the offsets committed to zookeeper of a kafka topic when we consumer it using "storm-kafka" integration & is it possbile to edit the offset committed to zookeeper to re-consume the topic messages again?

And can someone help me identify,

Which class of "storm-kafka" does actually write/commits the offset to zookeeper ? &

which class actually fetches the last committed offset when we re-run a storm topology with same client id ?

Thanks a lot.

Re: Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

pshah — Thu, 12 May 2016 05:22:51 GMT

Hi @Raja A Yes, it should be possible. As far as editing is concerned, it is advisable to not do it while topology is running. There is also an option to tell storm-kafka to not use zookeeper offsets. Which version of hdp are you wanting to look at the classes handling the writing and fetching offsets from zookeeper?

Re: Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

raja_aravapalli — Thu, 12 May 2016 23:00:00 GMT

Thanks for the response @pshah i am using the storm version, 0.9.3.2.2.4.0-2633

can you pls help me with what classes in this version are doing this fetching offsets/editing them into zookeeper.

Thanks very much.

My whole intension is to see the committed offsets and edit them as necessary.

Also, in which version of Storm i can say to storm to not use the zookeeper for offsets ?

Re: Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

pshah — Fri, 13 May 2016 00:40:32 GMT

@Raja A https://github.com/hortonworks/storm/blob/HDP-2.2.4.0-tag/external/storm-kafka/src/jvm/storm/kafka/PartitionManager.java is the file where its read and written. The zk path where it writes depends on your SpoutConfig. It is the class at https://github.com/hortonworks/storm/blob/HDP-2.2.4.0-tag/external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java

You can check out the method committedPath at https://github.com/hortonworks/storm/blob/HDP-2.2.4.0-tag/external/storm-kafka/src/jvm/storm/kafka/PartitionManager.java#L242

Re: Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

raja_aravapalli — Fri, 13 May 2016 01:39:08 GMT

Thanks very much @pshah, i will check those classes!! But, unfortunately, that links you shared navigating me to "page not found"!

But i can still check the classes by decompiling my storm version jars....!!

Re: Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

pshah — Fri, 13 May 2016 02:21:25 GMT

@Raja A Are you logged in to github ?

Re: Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

raja_aravapalli — Tue, 17 May 2016 00:52:41 GMT

with the help of the classes you mentioned i am able to check the committed kafka offsets!!

I am trying to write an application now to be able to edit the stored offsets!!

thanks a lot for your help!! @pshah

Re: Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

raja_aravapalli — Tue, 17 May 2016 00:54:00 GMT

I have a github account, but not really active on itt!! please let me know what information you are looking for ? @pshah

Re: Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

pshah — Tue, 17 May 2016 01:14:08 GMT

@Raja A To be able to look at those links you need to be logged in to your github account and have access to those repositories. Can you login and check?

Re: Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

raja_aravapalli — Tue, 17 May 2016 03:16:47 GMT

@pshah

okay will check by logging in.

Anyways i decompiled in editor and verified the code, and i am good!

Do we have any sample application that does the editing of stored offsets in zookeer ?

Re: Ability to Edit/Fetch the kafka offsets stored in zookeeper - "storm-kafka" integration library!!

pshah — Tue, 17 May 2016 03:17:12 GMT

@Raja A Not that i know of. You can edit the offsets using command line zookeeper client. In a test or dev setup this should be fine. I would not advise it in a production environment unless you know for sure that updating the offsets will not cause storm application to skip data for processing.