Support Questions

Find answers, ask questions, and share your expertise

Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

1628-spark15.png

1629-spark15-zeppelin.png

1 ACCEPTED SOLUTION

Zeppelin ships with Spark embedded with it. If you follow the Zeppelin tech preview link Neeraj pointed you can get it to work with Spark 1.5.x in HDP 2.3.4.

In Apache, Zeppelin released a version compatible with Spark 1.6 on Jan 22nd. You can get the binary of that Zeppelin from https://zeppelin.incubator.apache.org/download.html and follow the Zeppelin TP http://hortonworks.com/hadoop-tutorial/apache-zeppelin/ to get it work with Spark 1.6 Tech Preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

View solution in original post

13 REPLIES 13

Mentor

That's strange, do you have something in your classpatg referencing old Spark libs? @vbhoomireddy @vshukla

Explorer

Can you check what the SPARK_HOME value is in zeppelin-env.sh?

Expert Contributor

Interesting, my SPARK_HOME is not defined in that file. I have only that comment:

# export SPARK_HOME # (required) When it is defined, load it instead of Zeppelin embedded Spark libraries

Which explains why we use the Spark libraries in the zeppelin jar instead of the one defined in spark.yarn.jar ?

Explorer

have you tried to set SPARK_HOME? I think it most likely solves your problem

for more info take a look at this link, scroll down to the Configure section: https://github.com/apache/incubator-zeppelin

Expert Contributor

I have the same behaviour on my Sandbox (HDP2.3.4). This seems strange because the version number in spark.yarn.jar and in spark.home seemed to be totally bypassed.

If you look at the jar zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar inside <ZEPPELIN-HOME>/interpreter/spark, and if you extract the file META-INF/maven/org.apache.zeppelin/zeppelin-spark/pom.xml, you'll see this:

<spark.version>1.4.1</spark.version>

Mentor

@Sourygna Luangsay sandbox came out before spark 1.5.2 so its expected. Just make sure classpath is not pointibg to old jars

Expert Contributor

Actually, I upgraded my Sandbox to the last version of HDP.

When I do a "locate" on my Sandbox, I no longer find a reference to any spark-1.4.1 jar, only 1.5.2 jars.

Mentor

@Sourygna Luangsay and it still says sc.version 1.4.1? can you go into admin page on ambari and doublecheck that you completed the upgrade?

Zeppelin ships with Spark embedded with it. If you follow the Zeppelin tech preview link Neeraj pointed you can get it to work with Spark 1.5.x in HDP 2.3.4.

In Apache, Zeppelin released a version compatible with Spark 1.6 on Jan 22nd. You can get the binary of that Zeppelin from https://zeppelin.incubator.apache.org/download.html and follow the Zeppelin TP http://hortonworks.com/hadoop-tutorial/apache-zeppelin/ to get it work with Spark 1.6 Tech Preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

@Sourygna Luangsay

Please follow the tutorial step by step and it will help to resolve the issue

@Sourygna Luangsay It seems like you took HDP 2.3.2 sandbox (which comes with Spark 1.4.1) and upgraded to 2.3.4 (which has spark 1.5.1). However Zeppelin on 2.3.2 was compiled with Spark 1.4.1 - which is why it outputs sc.version =1.4.1

Since Zeppelin on 2.3.2 sandbox was deployed via the Ambari service, you can just follow steps below to delete it:

https://github.com/hortonworks-gallery/ambari-zeppelin-service#remove-zeppelin-service

Then follow the steps here to re-install it and it should install Zeppelin bits compiled with Spark 1.5.1

https://github.com/hortonworks-gallery/ambari-zeppelin-service#setup-the-ambari-service

Mentor

@vbhoomireddy are you still having issues with this? Can you accept the best answer or provide your own solution?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.