Kafka topic partition has unexpected offset[13]..last committed offset is [57]

[#1]

I am using metron to parse grok pattern. Even though on metron UI it seems it is able to parse the logs(its showing values after validation at UI) but in storm no error is coming and its not writing to enrichment topic...when i try to look further into it i come to know that some issue is there while consuming or producing kafka logs.

Its giving as "partition{syslog_withID-0} has unexpected offset[13] current committed offset is [57]"

I just have basic understanding about kafka but not in much depth. Would you please let me know how to solve this issue.

I have just one default partition syslog_withID-0...please help me out with this...

[#2]

I'm expecting the same issue. Did you manage to solve it? Any feedback is highly appreciated.

[#3]

Hi @Vinoth Krish and @Michael Hidalgo,

Question: Do you see indices being written into ES (or Solr) for your parser?

Here are few things you can check on your setup.

a) Check the offsets on the kafka topics for your topic using the command:

source /etc/default/metron
/usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --describe --group <the-topic-group> --bootstrap-server $BROKERLIST --new-consumer

You can repeat the above by substituting the group with the 'enrichment', 'indexing-ra' and so on. This will tell you the consumed offsets, current offset, lag and so on.

From your error, I am guessing there could be an issue with the current offset and the log-end-offset values.

b) Check your Kafka Enrichment offset strategy settings in Ambari UI, under Services -> Metron -> Configs -> Enrichment tab. One thing you can try for troubleshooting is to set this to 'EARLIEST' which allows all the tuples in the topic to be processed from the beginning .

[#4]

• Which version of HDP are you using?
• How many workers, spouts and ackers did you configure for you parser topology?