Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

avatar
Contributor

1628-spark15.png

1629-spark15-zeppelin.png

1 ACCEPTED SOLUTION

avatar

Zeppelin ships with Spark embedded with it. If you follow the Zeppelin tech preview link Neeraj pointed you can get it to work with Spark 1.5.x in HDP 2.3.4.

In Apache, Zeppelin released a version compatible with Spark 1.6 on Jan 22nd. You can get the binary of that Zeppelin from https://zeppelin.incubator.apache.org/download.html and follow the Zeppelin TP http://hortonworks.com/hadoop-tutorial/apache-zeppelin/ to get it work with Spark 1.6 Tech Preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

View solution in original post

13 REPLIES 13

avatar
Master Mentor

That's strange, do you have something in your classpatg referencing old Spark libs? @vbhoomireddy @vshukla

avatar
Contributor

Can you check what the SPARK_HOME value is in zeppelin-env.sh?

avatar
Super Collaborator

Interesting, my SPARK_HOME is not defined in that file. I have only that comment:

# export SPARK_HOME # (required) When it is defined, load it instead of Zeppelin embedded Spark libraries

Which explains why we use the Spark libraries in the zeppelin jar instead of the one defined in spark.yarn.jar ?

avatar
Contributor

have you tried to set SPARK_HOME? I think it most likely solves your problem

for more info take a look at this link, scroll down to the Configure section: https://github.com/apache/incubator-zeppelin

avatar
Super Collaborator

I have the same behaviour on my Sandbox (HDP2.3.4). This seems strange because the version number in spark.yarn.jar and in spark.home seemed to be totally bypassed.

If you look at the jar zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar inside <ZEPPELIN-HOME>/interpreter/spark, and if you extract the file META-INF/maven/org.apache.zeppelin/zeppelin-spark/pom.xml, you'll see this:

<spark.version>1.4.1</spark.version>

avatar
Master Mentor

@Sourygna Luangsay sandbox came out before spark 1.5.2 so its expected. Just make sure classpath is not pointibg to old jars

avatar
Super Collaborator

Actually, I upgraded my Sandbox to the last version of HDP.

When I do a "locate" on my Sandbox, I no longer find a reference to any spark-1.4.1 jar, only 1.5.2 jars.

avatar
Master Mentor

@Sourygna Luangsay and it still says sc.version 1.4.1? can you go into admin page on ambari and doublecheck that you completed the upgrade?

avatar
Master Mentor