- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Issue when deploying a new parser topology in Storm with KAFKA (Offset lags for kafka not supported for older versions)
- Labels:
-
Apache Metron
-
Apache Storm
Created on ‎11-16-2016 02:51 PM - edited ‎08-19-2019 04:48 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I followed the steps to deploy a new data source in Metron but I am blocked to the step : deploy the new parser topology because I noticed an error in STORM Topology complaining about KAFKA version issue
After I submitted the squid parder topogoly to HOST $HOST_WITH_ENRICHMENT_TAG as root.
I went o to the Storm UI and I noticed that the new "squid" topology was an error :
Topology spouts lag error
Id | Type | Message |
---|---|---|
kafkaSpout | KAFKA | Offset lags for kafka not supported for older versions. Please update kafka spout to latest version. |
Did you see this error before ?
What do I need to do to solve it ?
Thanks for your help
Maxime
Created ‎11-28-2016 09:36 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Metron 0.2.1BETA is based on HDP2.4, and storm 0.10.x
For HDP 2.5 you want to be using Metron 0.3.0, which has just been released. This will be built against the storm version your specified.
Created ‎11-17-2016 10:04 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
HDP 2.5 is actually based on Apache Storm 1.1.0-SNAPSHOT which means that some features are not yet available to official Apache Storm releases. "Topology Spout Lag" is one of them.
You can just ignore that error since it doesn't affect topology, or change the version of 'storm-kafka' to 1.0.1.2.5.0.0-1245 (NOTE: Hortonworks version) to make it work.
Hope this helps.
Created ‎11-17-2016 10:04 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Here some info about my SW versions :
METRON_VERSION=0.2.1BETA
Storm version : 1.0.1.2.5.0.0-1245
Created ‎11-28-2016 09:36 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Metron 0.2.1BETA is based on HDP2.4, and storm 0.10.x
For HDP 2.5 you want to be using Metron 0.3.0, which has just been released. This will be built against the storm version your specified.
Created ‎11-30-2016 03:59 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi simon, I upgraded Meton to 0.3.0 2 weeks ago and since my issue is gone.
thank you
Maxime
Created ‎03-20-2017 12:25 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi !
We are running Storm 1.0.1.2.5.0.0-1245 and Hadoop 2.5.0.0-1245, we get the same issue, any hint ?
Thanks for your help
Regards,
Vince
Created ‎03-20-2017 10:29 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Vincent.
What is the version of storm-kafka that you are using to build your topology? Can you try to upgrade it to the same version as mentioned by @Jungtaek Lim above?
