Support Questions
Find answers, ask questions, and share your expertise

java.lang.NoSuchMethodException exception with CDH 5.7.0, Apache Spark 1.6 and Zeppelin 0.6.0

Highlighted

java.lang.NoSuchMethodException exception with CDH 5.7.0, Apache Spark 1.6 and Zeppelin 0.6.0

Explorer

I running CDH 5.7.0 as a Virtual Box image on my iMac.

 

[cloudera@quickstart ~]$ spark-shell

-bash: spark-shell: command not found

[cloudera@quickstart ~]$ export PATH=/opt/cloudera/parcels/CDH/bin:$PATH

[cloudera@quickstart ~]$ spark-shell

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel).

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0

      /_/

 

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_67)

Type in expressions to have them evaluated.

Type :help for more information.

Spark context available as sc (master = yarn-client, app id = application_1462242771224_0006).

SQL context available as sqlContext.

 

scala>

 

 

 

[cloudera@quickstart incubator-zeppelin]$ export PATH=/opt/cloudera/parcels/CDH/bin:$PATH

[cloudera@quickstart incubator-zeppelin]$ hadoop version

Hadoop 2.6.0-cdh5.7.0

Subversion http://github.com/cloudera/hadoop -r c00978c67b0d3fe9f3b896b5030741bd40bf541a

Compiled by jenkins on 2016-03-23T18:36Z

Compiled with protoc 2.5.0

From source with checksum b2eabfa328e763c88cb14168f9b372

This command was run using /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/hadoop-common-2.6.0-cdh5.7.0.jar

[cloudera@quickstart incubator-zeppelin]$

 

I have compiled Zeppelin using the command 

 

[cloudera@quickstart incubator-zeppelin]$ mvn clean package -Pspark-1.6 -Ppyspark -Dhadoop.version=2.6.0-cdh5.7.0 -Phadoop-2.6 -Pyarn -Pvendor-repo -DskipTests

 

 

When I am trying to run the Zeppelin tutorial code 

 

import org.apache.commons.io.IOUtils
import java.net.URL
import java.nio.charset.Charset

// Zeppelin creates and injects sc (SparkContext) and sqlContext (HiveContext or SqlContext)
// So you don't need create them manually

// load bank data
val bankText = sc.parallelize(
    IOUtils.toString(
        new URL("https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv"),
        Charset.forName("utf8")).split("\n"))

case class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer)

val bank = bankText.map(s => s.split(";")).filter(s => s(0) != "\"age\"").map(
    s => Bank(s(0).toInt, 
            s(1).replaceAll("\"", ""),
            s(2).replaceAll("\"", ""),
            s(3).replaceAll("\"", ""),
            s(5).replaceAll("\"", "").toInt
        )
).toDF()
bank.registerTempTable("bank")

 

I am getting error

java.lang.NoSuchMethodException: org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.classServerUri()
at java.lang.Class.getMethod(Class.java:1665)
at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:275)
at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:150)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:525)
at org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:74)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:92)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:345)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
 
 
ERROR
Took 8 seconds
 
How can I fix the issue?
6 REPLIES 6
Highlighted

Re: java.lang.NoSuchMethodException exception with CDH 5.7.0, Apache Spark 1.6 and Zeppelin 0.6.0

Master Collaborator

This method was removed in Spark 2.0, but still is in the upstream 1.6 branch. It was removed in https://github.com/apache/spark/commit/4a46b8859d3314b5b45a67cdc5c81fecb6e9e78c#diff-0a5ec7561c8c486...

 

I think CDH 5.7 actually back-ported this change on top of Spark 1.6.x then, though that usually means it should have also been back-ported to the 1.6 branch upstream. Let me see if Marcelo can weigh in on what to do with that. 

 

If that's true I'm not sure exactly what it means for Zeppelin since it has to compile vs a later Spark version then that does not have this method.

 

Highlighted

Re: java.lang.NoSuchMethodException exception with CDH 5.7.0, Apache Spark 1.6 and Zeppelin 0.6.0

Explorer

Any update?

Highlighted

Re: java.lang.NoSuchMethodException exception with CDH 5.7.0, Apache Spark 1.6 and Zeppelin 0.6.0

Master Collaborator

That's the update. You may have to modify Zeppelin or build it with a different profile to get this to work. It's accessing a non-public API so YMMV. Zeppelin is not supported.

Highlighted

Re: java.lang.NoSuchMethodException exception with CDH 5.7.0, Apache Spark 1.6 and Zeppelin 0.6.0

Explorer

When you say different profile what do you mean by that?

 

Highlighted

Re: java.lang.NoSuchMethodException exception with CDH 5.7.0, Apache Spark 1.6 and Zeppelin 0.6.0

Master Collaborator

Here I'm referring to Zeppelin, since you showed it has profiles for various Spark versions. It's possible that a "Spark 2" profile actually works. But, I'm only guessing.

Highlighted

Re: java.lang.NoSuchMethodException exception with CDH 5.7.0, Apache Spark 1.6 and Zeppelin 0.6.0

New Contributor

Looks like the original problem was resolved in this commit:

https://github.com/apache/zeppelin/commit/78c7b5567e7fb4985cecf147c39033c554dfc208

 

After the fix I am able to run simple spark commands on zeppelin, but sending lambdas is not working. I've opened the following issue with zeppelin and made a pull request:

https://issues.apache.org/jira/browse/ZEPPELIN-1518

https://github.com/apache/zeppelin/pull/1486