Storm: Kafka to HDFS - dependency jar issues

raja_aravapalli — Sat, 17 Sep 2016 13:40:49 GMT

I have built a storm topology, that consumes data from kafka and writes into hdfs.

When using storm & kafka dependencies like below,

<storm.version>0.10.0.2.3.4.0-3485</storm.version>
<kafka.version>0.8.2.1</kafka.version>

topology is failing with below stated exception:

**java.lang.NoSuchMethodError: kafka.javaapi.consumer.SimpleConsumer.<init>(Ljava/lang/String;IIILjava/lang/String;Ljava/lang/String;)V at storm.kafka.DynamicPartitionConnections.register(DynamicPartitionConnections.java:60) at storm.kafka.PartitionManager.<init>(PartitionManager.java:66) at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:138) at backtype.storm.daemon.executor$fn__7098$fn__7113$fn__7142.invoke(executor.clj:596) at backtype.storm.util$async_loop$fn__543.invoke(util.clj:475) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)**

But, When using storm & kafka dependencies like below topology is running fine and downloading messages and writing to hdfs , without any issues!!

<storm.version>0.9.3.2.2.4.0-2633</storm.version>
<kafka.version>0.8.2.1</kafka.version>

Can someone please help what is the causing the issue and help me fix the issue !

Re: Storm: Kafka to HDFS - dependency jar issues

rajkumar_singh — Sun, 18 Sep 2016 08:02:19 GMT

@Raja A storm 0.10.0.2.3.4.0-3485 is build with kafka 8.2.2 where SimpleConsumer expect argument String,Int,Int,Int,String,String while 0.9.3.2.2.4.0-2633 is build with 0.8.1.1 where simpleconsumer expect args String,Int,Int,Int,String thats why you are getting NoSuchMethodError.

question Storm: Kafka to HDFS - dependency jar issues in Archives of Support Questions (Read Only)

Storm: Kafka to HDFS - dependency jar issues

Re: Storm: Kafka to HDFS - dependency jar issues