issue with storm Kafka Spout

[#1]

Hi all,

we're having some issues with a Storm topology (HDP 2.5), in which we have a Kafka Spout.

From the Storm UI we see that there is the following exception:

 java.lang.NoSuchMethodError: 
org.apache.curator.utils.ZKPaths.mkdirs(Lorg/apache/zookeeper/ZooKeeper;Ljava/lang/String;ZLorg/apache/curator/utils/InternalACLProvider;Z)V

	at 
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:727)

	at 
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:704)

	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
	at 
org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:701)

	at 
org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:477)

	at 
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:467)

	at 
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)

	at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:76)
	at org.apache.storm.kafka.ZkState.writeJSON(ZkState.java:70)
	at 
org.apache.storm.kafka.PartitionManager.commit(PartitionManager.java:312)

	at org.apache.storm.kafka.KafkaSpout.commit(KafkaSpout.java:236)
	at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:156)
	at 
org.apache.storm.daemon.executor$fn__6503$fn__6518$fn__6549.invoke(executor.clj:651)

	at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.lang.Thread.run(Thread.java:745)

While using HDP 2.4 we hadn't this issue.

Anyone knows why it happens?

Thank you very much

[#2]

@Sriharsha Chintalapani @tgoetz

[#3 · Accepted solution]

@Davide Vergari This might be the side effect of several things. We did a shading of all the storm dependencies so that topologies can bring their own version of common dependencies and storm libs wouldn't conflict with the user's topology dependencies.

easy fix would be to add to your topology following dependency

<dependency>                
  <groupId>org.apache.curator</groupId>
  <artifactId>curator-framework</artifactId>              
  <version>2.10.0</version>                
   <exclusions>                   
    <exclusion>                       
     <groupId>log4j</groupId>                        
     <artifactId>log4j</artifactId>                    
    </exclusion>                    
    <exclusion>                        
     <groupId>org.jboss.netty</groupId>                        
     <artifactId>netty</artifactId>                    
    </exclusion>                
  </exclusions>            
</dependency>

[#4]

@Sriharsha Chintalapani - We had a call on this issue last month through hortonworks case ticket and you suggested to add this dependency along with other changes. I added it and it was working fine. However, removing this dependency also worked.

Now from past 2 weeks, we came across the same exception twice. First time we resolved this issue by creating a new kafka topic and changing the consumer group name. Today, even that is not working. We keep getting the same exception regardless of curator jar being there or not.

Any pointers ?

******************************************************************************************

Unable to get offset lags for kafka. Reason: java.lang.NullPointerException at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:269) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:127)

**************************************************************************************************

java.lang.NoSuchMethodError: org.apache.curator.utils.ZKPaths.mkdirs(Lorg/apache/zookeeper/ZooKeeper;Ljava/lang/String;ZLorg/apache/curator/utils/InternalACLProvider;Z)V at org.apache.curator.framewo

Regards

Shekhar Suman

[#5]

Helpful to me in my HDP 2.5.3 cluster!!

[#6]

Faced the same issue on HDP 2.5.3. This issue comes if apache curator jars version is mismatched.

Earlier in my implementation curator-farmework, curator-client and curator-reciepe jar version were not same. So corrected them to 2.7.1 and it is working fine for me.