Support Questions
Find answers, ask questions, and share your expertise

Error connecting to kafka from Storm

Error connecting to kafka from Storm

Explorer

I am using Hortonworks HDP and trying to run a storm topology but getting this error. Look like ZooKeeper having some trouble. Spend the whole day trying to figure out this problem. note that i compile and make jar on windows and run in the hortonworks sandbox. I copied the whole code from sample

Shahzad

19232 [Thread-6] INFO  backtype.storm.config - SET worker-user 2626e3be-568c-4d9e-97fb-26b3fd07f8a1
20052 [Thread-10-kafkaSpout] ERROR backtype.storm.util - Async loop died!
java.lang.NoSuchMethodError: org.apache.zookeeper.server.quorum.flexible.QuorumMaj.<init>(Ljava/util/Map;)V
        at org.apache.curator.framework.imps.EnsembleTracker.<init>(EnsembleTracker.java:54) ~[curator-framework-3.0.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.<init>(CuratorFrameworkImpl.java:156) ~[curator-framework-3.0.0.jar:na]
        at org.apache.curator.framework.CuratorFrameworkFactory$Builder.build(CuratorFrameworkFactory.java:136) ~[curator-framework-3.0.0.jar:na]
        at org.apache.curator.framework.CuratorFrameworkFactory.newClient(CuratorFrameworkFactory.java:107) ~[curator-framework-3.0.0.jar:na]
        at storm.kafka.ZkState.newCurator(ZkState.java:45) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
        at storm.kafka.ZkState.<init>(ZkState.java:61) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
        at storm.kafka.KafkaSpout.open(KafkaSpout.java:85) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.daemon.executor$fn__4949$fn__4964.invoke(executor.clj:542) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
        at backtype.storm.util$async_loop$fn__452.invoke(util.clj:463) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
20067 [Thread-10-kafkaSpout] ERROR backtype.storm.daemon.executor -
java.lang.NoSuchMethodError: org.apache.zookeeper.server.quorum.flexible.QuorumMaj.<init>(Ljava/util/Map;)V
        at org.apache.curator.framework.imps.EnsembleTracker.<init>(EnsembleTracker.java:54) ~[curator-framework-3.0.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.<init>(CuratorFrameworkImpl.java:156) ~[curator-framework-3.0.0.jar:na]
        at org.apache.curator.framework.CuratorFrameworkFactory$Builder.build(CuratorFrameworkFactory.java:136) ~[curator-framework-3.0.0.jar:na]
        at org.apache.curator.framework.CuratorFrameworkFactory.newClient(CuratorFrameworkFactory.java:107) ~[curator-framework-3.0.0.jar:na]
        at storm.kafka.ZkState.newCurator(ZkState.java:45) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
        at storm.kafka.ZkState.<init>(ZkState.java:61) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
        at storm.kafka.KafkaSpout.open(KafkaSpout.java:85) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.daemon.executor$fn__4949$fn__4964.invoke(executor.clj:542) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
        at backtype.storm.util$async_loop$fn__452.invoke(util.clj:463) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
20120 [Thread-10-kafkaSpout] ERROR backtype.storm.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
        at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:322) [storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
        at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
        at backtype.storm.daemon.worker$fn__5425$fn__5426.invoke(worker.clj:491) [storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
        at backtype.storm.daemon.executor$mk_executor_data$fn__4850$fn__4851.invoke(executor.clj:245) [storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
        at backtype.storm.util$async_loop$fn__452.invoke(util.clj:475) [storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]


3 REPLIES 3
Highlighted

Re: Error connecting to kafka from Storm

Mentor

@Shahzad Aslam please post your pom file

Highlighted

Re: Error connecting to kafka from Storm

Cloudera Employee

@Shahzad Aslam It's duplicated question

https://community.hortonworks.com/questions/11061/storm-bolt-is-not-connected-with-kafka-look-like-s...

And I've left answer about previous question. Did you take a look?

Highlighted

Re: Error connecting to kafka from Storm

Guru

The no such method exception generally points to class conflict and stems from having the wrong versions in your pom.xml. Check your HDP stack and versions in ambari and make sure that your Zookkeeper, Storm, and Kafka dependencies are the correct versions.

<dependencies>

    <dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-core</artifactId>

		<version>0.10.0</version>

		<scope>provided</scope>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

    </dependency>

    <dependency>

		<groupId>org.apache.zookeeper</groupId>

		<artifactId>zookeeper</artifactId>

		<version>3.4.6</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-hdfs</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-kafka</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

	<groupId>org.apache.kafka</groupId>

	<artifactId>kafka_2.10</artifactId>

	<version>0.8.2.2</version>

	<exclusions>

                <exclusion>

                    <groupId>org.apache.zookeeper</groupId>

                    <artifactId>zookeeper</artifactId>

                </exclusion>

                <exclusion>

                    <groupId>log4j</groupId>

                    <artifactId>log4j</artifactId>

                </exclusion>

            </exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-hbase</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

    <dependency>

		<groupId>org.apache.hbase</groupId>

		<artifactId>hbase-client</artifactId>

		<version>1.1.1</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>	

	</dependency>