question Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1 in Archives of Support Questions (Read Only)

Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

vbhoomireddy — Mon, 19 Aug 2019 10:50:42 GMT

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

aervits — Fri, 29 Jan 2016 04:33:12 GMT

That's strange, do you have something in your classpatg referencing old Spark libs? @vbhoomireddy @vshukla

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

mahan — Fri, 29 Jan 2016 06:13:00 GMT

Can you check what the SPARK_HOME value is in zeppelin-env.sh?

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

sluangsay — Fri, 29 Jan 2016 06:13:26 GMT

I have the same behaviour on my Sandbox (HDP2.3.4). This seems strange because the version number in spark.yarn.jar and in spark.home seemed to be totally bypassed.

If you look at the jar zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar inside <ZEPPELIN-HOME>/interpreter/spark, and if you extract the file META-INF/maven/org.apache.zeppelin/zeppelin-spark/pom.xml, you'll see this:

<spark.version>1.4.1</spark.version>

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

sluangsay — Fri, 29 Jan 2016 06:17:01 GMT

Interesting, my SPARK_HOME is not defined in that file. I have only that comment:

# export SPARK_HOME # (required) When it is defined, load it instead of Zeppelin embedded Spark libraries

Which explains why we use the Spark libraries in the zeppelin jar instead of the one defined in spark.yarn.jar ?

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

aervits — Fri, 29 Jan 2016 06:26:37 GMT

@Sourygna Luangsay sandbox came out before spark 1.5.2 so its expected. Just make sure classpath is not pointibg to old jars

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

sluangsay — Fri, 29 Jan 2016 06:48:31 GMT

Actually, I upgraded my Sandbox to the last version of HDP.

When I do a "locate" on my Sandbox, I no longer find a reference to any spark-1.4.1 jar, only 1.5.2 jars.

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

aervits — Fri, 29 Jan 2016 06:50:25 GMT

@Sourygna Luangsay and it still says sc.version 1.4.1? can you go into admin page on ambari and doublecheck that you completed the upgrade?

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

mahan — Fri, 29 Jan 2016 23:09:56 GMT

have you tried to set SPARK_HOME? I think it most likely solves your problem

for more info take a look at this link, scroll down to the Configure section: https://github.com/apache/incubator-zeppelin

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

nsabharwal — Fri, 29 Jan 2016 23:20:49 GMT

@Sourygna Luangsay

Please follow this step by step http://hortonworks.com/hadoop-tutorial/apache-zeppelin/

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

vshukla — Sat, 30 Jan 2016 03:56:54 GMT

Zeppelin ships with Spark embedded with it. If you follow the Zeppelin tech preview link Neeraj pointed you can get it to work with Spark 1.5.x in HDP 2.3.4.

In Apache, Zeppelin released a version compatible with Spark 1.6 on Jan 22nd. You can get the binary of that Zeppelin from https://zeppelin.incubator.apache.org/download.html and follow the Zeppelin TP http://hortonworks.com/hadoop-tutorial/apache-zeppelin/ to get it work with Spark 1.6 Tech Preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

nsabharwal — Sat, 30 Jan 2016 06:40:27 GMT

@Sourygna Luangsay

Please follow the tutorial step by step and it will help to resolve the issue

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

abajwa — Mon, 01 Feb 2016 14:59:39 GMT

@Sourygna Luangsay It seems like you took HDP 2.3.2 sandbox (which comes with Spark 1.4.1) and upgraded to 2.3.4 (which has spark 1.5.1). However Zeppelin on 2.3.2 was compiled with Spark 1.4.1 - which is why it outputs sc.version =1.4.1

Since Zeppelin on 2.3.2 sandbox was deployed via the Ambari service, you can just follow steps below to delete it:

https://github.com/hortonworks-gallery/ambari-zeppelin-service#remove-zeppelin-service

Then follow the steps here to re-install it and it should install Zeppelin bits compiled with Spark 1.5.1

https://github.com/hortonworks-gallery/ambari-zeppelin-service#setup-the-ambari-service

Re: Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

aervits — Tue, 02 Feb 2016 10:28:08 GMT

@vbhoomireddy are you still having issues with this? Can you accept the best answer or provide your own solution?