question Re: "ForceFromStart" option in Storm-Kafka 1.0 in Support Questions

"ForceFromStart" option in Storm-Kafka 1.0

maldarizzi — Thu, 08 Sep 2016 22:38:08 GMT

Hi All,

We have upgraded our environment from HDP 2.3.4 to HDP 2.4 and so review Storm topologies in order to use 1.0.1 version and new features. In old version, storm-kafka library (0.9.2) included the option "forceFromStart" in SpoutConfig that restore data from beginning (if set to true) or get only current value (if set to false) by kafka.

In last version the option is removed and I see that the spout get always data from beginning.

How we can replace the functionality in new version? I mean, I want get only the current value from spout, how I can proceed to do it?

Thanks in advance, Giuseppe

Re: "ForceFromStart" option in Storm-Kafka 1.0

aervits — Fri, 09 Sep 2016 09:13:34 GMT

Storm 1.0.1 is only available in HDP 2.5 so I'm wondering whether you have issues with dependencies. according to the latest documentation for 2.5 forceFeomStart is still there and defaults to false. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_storm-component-guide/content/storm-kafkaspout-config-core.html

Re: "ForceFromStart" option in Storm-Kafka 1.0

maldarizzi — Fri, 09 Sep 2016 15:13:46 GMT

Thank you Artem! Yes, I'm working with 2.5, I was wrong to write above...

The dependencies are ok but the KafkaConfig don't have "forceFromStart" properties. In fact, I've checked the KafkaConfig sources and it's missing, please see

https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/KafkaConfig.java

So, it could be a refuse in documentation. In this case, how I can use this properties if it is not present in kafkaConfig?

Re: "ForceFromStart" option in Storm-Kafka 1.0

aervits — Fri, 09 Sep 2016 21:26:02 GMT

I will follow up with engineering on this but for now, can you try the following

kafka.api.OffsetRequest.LatestTime()

that should only grab latest messages

Re: "ForceFromStart" option in Storm-Kafka 1.0

aervits — Fri, 09 Sep 2016 21:33:46 GMT

@Giuseppe Maldarizzi

looks like you're correct, forceFromStart is removed from Storm in favor of EarliestTime/LatestTime, it is a problem with documentation, I will speak to docs team to remove that and replace with correct info. The associated JIRAs are https://issues.apache.org/jira/browse/STORM-563 and STORM-650

Re: "ForceFromStart" option in Storm-Kafka 1.0

maldarizzi — Fri, 09 Sep 2016 22:31:34 GMT

Thanks you Artem for clarification, I will proceed using new approach.

Re: "ForceFromStart" option in Storm-Kafka 1.0

aervits — Fri, 09 Sep 2016 22:55:16 GMT

@Giuseppe Maldarizzi once you confirm it works, please accept the answer to close the thread.

Re: "ForceFromStart" option in Storm-Kafka 1.0

maldarizzi — Fri, 09 Sep 2016 22:57:32 GMT

Yes, it work, we can close the thread. Thanks again

Re: "ForceFromStart" option in Storm-Kafka 1.0

aervits — Sat, 10 Sep 2016 00:25:52 GMT

@Giuseppe Maldarizzi just heard back from engineering, also please look at ignoreZKOffsets parameter in place of forceFromStart, documentation will be updated. https://github.com/apache/storm/tree/master/external/storm-kafka#how-kafkaspout-stores-offsets-of-a-kafka-topic-and-recovers-in-case-of-failures

Re: "ForceFromStart" option in Storm-Kafka 1.0

maldarizzi — Mon, 12 Sep 2016 14:44:15 GMT

Perfect, I will try it. Thank you again