Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark 1.5 with Zeppelin - but sc.version prints 1.4.1

avatar
Contributor

1628-spark15.png

1629-spark15-zeppelin.png

1 ACCEPTED SOLUTION

avatar

Zeppelin ships with Spark embedded with it. If you follow the Zeppelin tech preview link Neeraj pointed you can get it to work with Spark 1.5.x in HDP 2.3.4.

In Apache, Zeppelin released a version compatible with Spark 1.6 on Jan 22nd. You can get the binary of that Zeppelin from https://zeppelin.incubator.apache.org/download.html and follow the Zeppelin TP http://hortonworks.com/hadoop-tutorial/apache-zeppelin/ to get it work with Spark 1.6 Tech Preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

View solution in original post

13 REPLIES 13

avatar

Zeppelin ships with Spark embedded with it. If you follow the Zeppelin tech preview link Neeraj pointed you can get it to work with Spark 1.5.x in HDP 2.3.4.

In Apache, Zeppelin released a version compatible with Spark 1.6 on Jan 22nd. You can get the binary of that Zeppelin from https://zeppelin.incubator.apache.org/download.html and follow the Zeppelin TP http://hortonworks.com/hadoop-tutorial/apache-zeppelin/ to get it work with Spark 1.6 Tech Preview (http://hortonworks.com/hadoop-tutorial/apache-spark-1-6-technical-preview-with-hdp-2-3/)

avatar
Master Mentor

@Sourygna Luangsay

Please follow the tutorial step by step and it will help to resolve the issue

avatar

@Sourygna Luangsay It seems like you took HDP 2.3.2 sandbox (which comes with Spark 1.4.1) and upgraded to 2.3.4 (which has spark 1.5.1). However Zeppelin on 2.3.2 was compiled with Spark 1.4.1 - which is why it outputs sc.version =1.4.1

Since Zeppelin on 2.3.2 sandbox was deployed via the Ambari service, you can just follow steps below to delete it:

https://github.com/hortonworks-gallery/ambari-zeppelin-service#remove-zeppelin-service

Then follow the steps here to re-install it and it should install Zeppelin bits compiled with Spark 1.5.1

https://github.com/hortonworks-gallery/ambari-zeppelin-service#setup-the-ambari-service

avatar
Master Mentor

@vbhoomireddy are you still having issues with this? Can you accept the best answer or provide your own solution?