question Problem: Zeppelin 0.6 and Spark 2 on HDP 2.5 in Archives of Support Questions (Read Only)

Problem: Zeppelin 0.6 and Spark 2 on HDP 2.5

chiru_tnk — Sun, 18 Aug 2019 04:28:50 GMT

Dear experts,

I am running HDP 2.5 sandbox on Azure with Zeppelin 0.6.0.2.5 and Spark 2.0.0.2.5. Zeppelin is able to recognize only the Spark 1 but not 2, could you please help on this ? Are the versions incompatible ? I tried creating a new interpreter as spark2 as in attached images but still not working.

Below are the installed spark versions in my sandbox

[root@sandbox spark2]# spark-shell --version Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set Spark1 will be picked by default Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.2 /_/ Type --help for more information. [root@sandbox spark2]# export SPARK_MAJOR_VERSION=2 [root@sandbox spark2]# spark-submit --version SPARK_MAJOR_VERSION is set to 2, using Spark2 Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.0.0.2.5.0.0-1245 /_/ Branch HEAD Compiled by user jenkins on 2016-08-26T02:52:23Z Revision c036d8e15666e914cd6aaf90ada0390619828b10 Url git@github.com:hortonworks/spark2.git Type --help for more information.

Re: Problem: Zeppelin 0.6 and Spark 2 on HDP 2.5

jan_dombrowicz — Fri, 24 Nov 2017 03:17:53 GMT

Hi @Chiranjeevi Nimmala

Indeed HDP 2.5 Sandbox and Spark2 with Zeppelin are not compatible.

Read this Blog article for more infos:
https://de.hortonworks.com/blog/try-latest-innovations-apache-spark-apache-zeppelin-hortonworks-2-5-sandbox/

If you don't want to download the newer Sandbox version, try this tutorial:
https://community.hortonworks.com/articles/53029/how-to-install-and-run-spark-20-on-hdp-25-sandbox.html

Re: Problem: Zeppelin 0.6 and Spark 2 on HDP 2.5

chiru_tnk — Fri, 24 Nov 2017 05:39:58 GMT

Thanks @Jan Dombrowicz ,i will use spark 1 with the zeppelin.

Re: Problem: Zeppelin 0.6 and Spark 2 on HDP 2.5

paulhernandez — Sat, 25 Nov 2017 15:26:00 GMT

Hi @Chiranjeevi Nimmala

you can have a look at my last blog post, it may help you: Installing Apache Zeppelin 0.7.3 in HDP 2.5.3 with Spark and Spark2 Interpreters

Re: Problem: Zeppelin 0.6 and Spark 2 on HDP 2.5

rajagopk — Thu, 01 Feb 2018 18:54:04 GMT

Hi @Paul:

Thank you for the blog. I am working on HDP 2.5.0 in Azure cloud. I am on mac logged into sandbox from terminal in mac.

I am not able to locate the install directory of zeppelin. Can you give some pointers?

Thanks

Krishna

Re: Problem: Zeppelin 0.6 and Spark 2 on HDP 2.5

paulhernandez — Thu, 01 Feb 2018 19:13:35 GMT

Hi @Krishnaswami Rajagopalan

I don't know exactly the detail of this Sandbox in the Azure Cloud.

Are you connecting to the Sandbox or to the docker container inside?

The docker container is where zeppelin and other services are located.

To connect to the docker container use the port 2222 in your SSH command.

Example: ssh root@127.0.0.1 -p 2222

It doesn't matter I guess if your cluster or sandbox is running on the cloud. You should be able to find zeppelin under /usr/hdp/current/zeppelin-server

Hope this helps.

BR. Paul