Support Questions

Find answers, ask questions, and share your expertise

issue with storm Kafka Spout

avatar
Expert Contributor

Hi all,

we're having some issues with a Storm topology (HDP 2.5), in which we have a Kafka Spout.

From the Storm UI we see that there is the following exception:

 java.lang.NoSuchMethodError: 
org.apache.curator.utils.ZKPaths.mkdirs(Lorg/apache/zookeeper/ZooKeeper;Ljava/lang/String;ZLorg/apache/curator/utils/InternalACLProvider;Z)V

	at 
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:727)

	at 
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:704)

	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
	at 
org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:701)

	at 
org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:477)

	at 
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:467)

	at 
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)

	at org.apache.storm.kafka.ZkState.writeBytes(ZkState.java:76)
	at org.apache.storm.kafka.ZkState.writeJSON(ZkState.java:70)
	at 
org.apache.storm.kafka.PartitionManager.commit(PartitionManager.java:312)

	at org.apache.storm.kafka.KafkaSpout.commit(KafkaSpout.java:236)
	at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:156)
	at 
org.apache.storm.daemon.executor$fn__6503$fn__6518$fn__6549.invoke(executor.clj:651)

	at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.lang.Thread.run(Thread.java:745)

While using HDP 2.4 we hadn't this issue.

Anyone knows why it happens?

Thank you very much

1 ACCEPTED SOLUTION

avatar

@Davide Vergari This might be the side effect of several things. We did a shading of all the storm dependencies so that topologies can bring their own version of common dependencies and storm libs wouldn't conflict with the user's topology dependencies.

easy fix would be to add to your topology following dependency

<dependency>                
  <groupId>org.apache.curator</groupId>
  <artifactId>curator-framework</artifactId>              
  <version>2.10.0</version>                
   <exclusions>                   
    <exclusion>                       
     <groupId>log4j</groupId>                        
     <artifactId>log4j</artifactId>                    
    </exclusion>                    
    <exclusion>                        
     <groupId>org.jboss.netty</groupId>                        
     <artifactId>netty</artifactId>                    
    </exclusion>                
  </exclusions>            
</dependency>

View solution in original post

5 REPLIES 5

avatar
Master Mentor

avatar

@Davide Vergari This might be the side effect of several things. We did a shading of all the storm dependencies so that topologies can bring their own version of common dependencies and storm libs wouldn't conflict with the user's topology dependencies.

easy fix would be to add to your topology following dependency

<dependency>                
  <groupId>org.apache.curator</groupId>
  <artifactId>curator-framework</artifactId>              
  <version>2.10.0</version>                
   <exclusions>                   
    <exclusion>                       
     <groupId>log4j</groupId>                        
     <artifactId>log4j</artifactId>                    
    </exclusion>                    
    <exclusion>                        
     <groupId>org.jboss.netty</groupId>                        
     <artifactId>netty</artifactId>                    
    </exclusion>                
  </exclusions>            
</dependency>

avatar
New Contributor

@Sriharsha Chintalapani - We had a call on this issue last month through hortonworks case ticket and you suggested to add this dependency along with other changes. I added it and it was working fine. However, removing this dependency also worked.

Now from past 2 weeks, we came across the same exception twice. First time we resolved this issue by creating a new kafka topic and changing the consumer group name. Today, even that is not working. We keep getting the same exception regardless of curator jar being there or not.

Any pointers ?

******************************************************************************************

Unable to get offset lags for kafka. Reason: java.lang.NullPointerException at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:269) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:127)

**************************************************************************************************

java.lang.NoSuchMethodError: org.apache.curator.utils.ZKPaths.mkdirs(Lorg/apache/zookeeper/ZooKeeper;Ljava/lang/String;ZLorg/apache/curator/utils/InternalACLProvider;Z)V at org.apache.curator.framewo

Regards

Shekhar Suman

avatar

Helpful to me in my HDP 2.5.3 cluster!!

avatar
New Contributor

Faced the same issue on HDP 2.5.3. This issue comes if apache curator jars version is mismatched.

Earlier in my implementation curator-farmework, curator-client and curator-reciepe jar version were not same. So corrected them to 2.7.1 and it is working fine for me.