Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎09-30-2018

Unable to resolve - NoSuchMethodError: kafka.utils.ZkUtils$.createPersistentPath

[ Edited ]

I've 3 node cluster setup using cloudera manager on CDH 5.14.2. My spark is running on Yarn.

 

I've a simple program which writes to zookeeper and and get the data back from zookeeper.

The program as follows: 

 

 

import kafka.utils.{ZKStringSerializer, ZkUtils}
import org.I0Itec.zkclient.ZkClient


object ZkutilsTest {
  def main(args: Array[String]): Unit = {
    val zkClient = new ZkClient("127.0.0.1", 10000, 10000, ZKStringSerializer)
    println("Value put to Zookeeper: Initial value is 1")
    ZkUtils.createPersistentPath(zkClient, "/testingZKUtils", "Initial value is 1")
    val (startDateFetch, _) = ZkUtils.readDataMaybeNull(zkClient, "/testingZKUtils")
    println("Value from Zookeeper: " + startDateFetch)
  }
}

my pom.xml have the following dependencies.

 

 

 

 <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka_2.10</artifactId>
            <version>1.6.0</version>
        </dependency>
    </dependencies>

 

While the compilation never gives exception. But when submitting the job to yarn-client, I get the following exception:

 

Exception in thread "main" java.lang.NoSuchMethodError: kafka.utils.ZkUtils$.createPersistentPath(Lorg/I0Itec/zkclient/ZkClient;Ljava/lang/String;Ljava/lang/String;)V
        at ZkutilsTest$.main(ZkutilsTest.scala:24)
        at ZkutilsTest.main(ZkutilsTest.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:730)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

My spark submit is: 

 

sudo spark-submit --master yarn-client --class ZkutilsTest target/SparkTest-1.0-SNAPSHOT-jar-with-dependencies.jar

 

I've tried changing the scope to provided in pom.xml still no luck.

 

Any help would be appreciated.

 

Announcements