Problem: Zeppelin 0.6 and Spark 2 on HDP 2.5

[#1]

Dear experts,

I am running HDP 2.5 sandbox on Azure with Zeppelin 0.6.0.2.5 and Spark 2.0.0.2.5. Zeppelin is able to recognize only the Spark 1 but not 2, could you please help on this ? Are the versions incompatible ? I tried creating a new interpreter as spark2 as in attached images but still not working.

Below are the installed spark versions in my sandbox

[root@sandbox spark2]# spark-shell --version Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set Spark1 will be picked by default Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.2 /_/ Type --help for more information. [root@sandbox spark2]# export SPARK_MAJOR_VERSION=2 [root@sandbox spark2]# spark-submit --version SPARK_MAJOR_VERSION is set to 2, using Spark2 Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.0.0.2.5.0.0-1245 /_/ Branch HEAD Compiled by user jenkins on 2016-08-26T02:52:23Z Revision c036d8e15666e914cd6aaf90ada0390619828b10 Url git@github.com:hortonworks/spark2.git Type --help for more information.

[#2 · Accepted solution]

Hi @Chiranjeevi Nimmala

Indeed HDP 2.5 Sandbox and Spark2 with Zeppelin are not compatible.

Read this Blog article for more infos:
https://de.hortonworks.com/blog/try-latest-innovations-apache-spark-apache-zeppelin-hortonworks-2-5-...

If you don't want to download the newer Sandbox version, try this tutorial:
https://community.hortonworks.com/articles/53029/how-to-install-and-run-spark-20-on-hdp-25-sandbox.h...

[#3]

Thanks @Jan Dombrowicz ,i will use spark 1 with the zeppelin.

[#4]

Hi @Chiranjeevi Nimmala

you can have a look at my last blog post, it may help you: Installing Apache Zeppelin 0.7.3 in HDP 2.5.3 with Spark and Spark2 Interpreters

[#5]

Hi @Paul:

Thank you for the blog. I am working on HDP 2.5.0 in Azure cloud. I am on mac logged into sandbox from terminal in mac.

I am not able to locate the install directory of zeppelin. Can you give some pointers?

Thanks

Krishna

[#6]

Hi @Krishnaswami Rajagopalan

I don't know exactly the detail of this Sandbox in the Azure Cloud.

Are you connecting to the Sandbox or to the docker container inside?

The docker container is where zeppelin and other services are located.

To connect to the docker container use the port 2222 in your SSH command.

Example: ssh root@127.0.0.1 -p 2222

It doesn't matter I guess if your cluster or sandbox is running on the cloud. You should be able to find zeppelin under /usr/hdp/current/zeppelin-server

Hope this helps.

BR. Paul