Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

[#1]

[#2 · Accepted solution]

Zeppelin ships with Spark embedded with it. If you follow the Zeppelin tech preview link Neeraj pointed you can get it to work with Spark 1.5.x in HDP 2.3.4.

In Apache, Zeppelin released a version compatible with Spark 1.6 on Jan 22nd. You can get the binary of that Zeppelin from https://zeppelin.incubator.apache.org/download.html and follow the Zeppelin TP http://hortonworks.com/hadoop-tutorial/apache-zeppelin/ to get it work with Spark 1.6 Tech Preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

[#3]

@Sourygna Luangsay

Please follow the tutorial step by step and it will help to resolve the issue

[#4]

@Sourygna Luangsay It seems like you took HDP 2.3.2 sandbox (which comes with Spark 1.4.1) and upgraded to 2.3.4 (which has spark 1.5.1). However Zeppelin on 2.3.2 was compiled with Spark 1.4.1 - which is why it outputs sc.version =1.4.1

Since Zeppelin on 2.3.2 sandbox was deployed via the Ambari service, you can just follow steps below to delete it:

https://github.com/hortonworks-gallery/ambari-zeppelin-service#remove-zeppelin-service

Then follow the steps here to re-install it and it should install Zeppelin bits compiled with Spark 1.5.1

https://github.com/hortonworks-gallery/ambari-zeppelin-service#setup-the-ambari-service

[#5]

@vbhoomireddy are you still having issues with this? Can you accept the best answer or provide your own solution?