https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-handle-kafka-common-OffsetOutOfRangeException-in/m-p/95544#M8894 <P>Make sure you set the following config in kafkaspout's Spoutconfig</P><PRE>spoutConfig.startOffsetTime = kafka.api.OffsetRequest.EarliestTime();</PRE><P><A href="https://github.com/apache/storm/tree/master/external/storm-kafka" target="_blank">https://github.com/apache/storm/tree/master/external/storm-kafka</A></P><P>Apart from that</P><P>1. Make sure you log.retention.hours is long enough to retain topic data</P><P>2. Check kafka topic offsets</P><P> bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hostname:6667 --topic topic_name --time -1</P><P>the above command will give you latest offset into kafka topic and now you need to check if storm kafkaspout is catching up.</P><P>2.1 login into zookeeper shell</P><P>2.2 ls /zkroot/id (zkroot is the one configured in spoutconfig and id from spoutconfig) as well</P><P>2.3 get /zkroot/id/topic_name/part_0 will give you a json structure with key "offset" this will tell you how far you read into topic and also how far you are behind reading the latest data.</P><P>If its too far apart and if log.retention.hours hit you kafkaspout might be requesting for older offset which might have been deleted.</P> Sat, 17 Oct 2015 06:27:49 GMT schintalapani 2015-10-17T06:27:49Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-handle-kafka-common-OffsetOutOfRangeException-in/m-p/95543#M8893 <P>I am getting following error with Storm topology failing to read the kafka as the data from kafka topic as data gets deleted based on retention(Size). Ideally we want to have storm to move (fast forward) to the first available message. Is there anyway to do this in Storm ?</P><P>[2015-10-13 14:12:28,204] ERROR [KafkaApi-2] Error when processing fetch request for partition [&lt;topic_name&gt;,0] offset 4231749539 from consumer with correlation id 0 (kafka.server.KafkaApis) kafka.common.OffsetOutOfRangeException: Request for offset 4231749539 but we only have log segments in the range 4255773954 to 4376049622. at kafka.log.Log.read(Log.scala:380) </P> Fri, 16 Oct 2015 17:52:53 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-handle-kafka-common-OffsetOutOfRangeException-in/m-p/95543#M8893 Jagatheeshr 2015-10-16T17:52:53Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-handle-kafka-common-OffsetOutOfRangeException-in/m-p/95544#M8894 <P>Make sure you set the following config in kafkaspout's Spoutconfig</P><PRE>spoutConfig.startOffsetTime = kafka.api.OffsetRequest.EarliestTime();</PRE><P><A href="https://github.com/apache/storm/tree/master/external/storm-kafka" target="_blank">https://github.com/apache/storm/tree/master/external/storm-kafka</A></P><P>Apart from that</P><P>1. Make sure you log.retention.hours is long enough to retain topic data</P><P>2. Check kafka topic offsets</P><P> bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list hostname:6667 --topic topic_name --time -1</P><P>the above command will give you latest offset into kafka topic and now you need to check if storm kafkaspout is catching up.</P><P>2.1 login into zookeeper shell</P><P>2.2 ls /zkroot/id (zkroot is the one configured in spoutconfig and id from spoutconfig) as well</P><P>2.3 get /zkroot/id/topic_name/part_0 will give you a json structure with key "offset" this will tell you how far you read into topic and also how far you are behind reading the latest data.</P><P>If its too far apart and if log.retention.hours hit you kafkaspout might be requesting for older offset which might have been deleted.</P> Sat, 17 Oct 2015 06:27:49 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-handle-kafka-common-OffsetOutOfRangeException-in/m-p/95544#M8894 schintalapani 2015-10-17T06:27:49Z