Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

[#1]

[#2]

That's strange, do you have something in your classpatg referencing old Spark libs? @vbhoomireddy @vshukla

[#3]

Can you check what the SPARK_HOME value is in zeppelin-env.sh?

[#4]

Interesting, my SPARK_HOME is not defined in that file. I have only that comment:

# export SPARK_HOME # (required) When it is defined, load it instead of Zeppelin embedded Spark libraries

Which explains why we use the Spark libraries in the zeppelin jar instead of the one defined in spark.yarn.jar ?

[#5]

have you tried to set SPARK_HOME? I think it most likely solves your problem

for more info take a look at this link, scroll down to the Configure section: https://github.com/apache/incubator-zeppelin

[#6]

I have the same behaviour on my Sandbox (HDP2.3.4). This seems strange because the version number in spark.yarn.jar and in spark.home seemed to be totally bypassed.

If you look at the jar zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar inside <ZEPPELIN-HOME>/interpreter/spark, and if you extract the file META-INF/maven/org.apache.zeppelin/zeppelin-spark/pom.xml, you'll see this:

<spark.version>1.4.1</spark.version>

[#7]

@Sourygna Luangsay sandbox came out before spark 1.5.2 so its expected. Just make sure classpath is not pointibg to old jars

[#8]

Actually, I upgraded my Sandbox to the last version of HDP.

When I do a "locate" on my Sandbox, I no longer find a reference to any spark-1.4.1 jar, only 1.5.2 jars.

[#9]

@Sourygna Luangsay and it still says sc.version 1.4.1? can you go into admin page on ambari and doublecheck that you completed the upgrade?

[#10]

@Sourygna Luangsay

Please follow this step by step http://hortonworks.com/hadoop-tutorial/apache-zeppelin/