Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is it possible to run Spark 2.0 version in Zeppelin on HDP 25 sandbox

Re: Is it possible to run Spark 2.0 version in Zeppelin on HDP 25 sandbox

New Contributor

I've restarted zeppelin, recreated notebook, verified that interpreter is enabled (by default it's always disabled) but have still same exception. Please, can you verify that below configuration is correct:

12302-zeppelin-configuration.png

Re: Is it possible to run Spark 2.0 version in Zeppelin on HDP 25 sandbox

Can you also post your %spark config snapshot side-by-side? Also Can you give this thing a try - using %spark2.spark instead of %spark2 - like this

%spark2.spark

spark.sparkContext.version

Re: Is it possible to run Spark 2.0 version in Zeppelin on HDP 25 sandbox

New Contributor

There are some more screens:

12394-zeppelin-spark-1.png

12395-zeppelin-spark-2.png

12396-zeppelin-spark-3.png

12397-zeppelin-spark-4.png

12398-zeppelin-spark-5.png

There are two things which looks interesting:

1) the output for interpreter spark2 looks different than it two other cases

2) spark2 is always by default off when I create new notebook. then I'm enabling it after I create notebook.

Re: Is it possible to run Spark 2.0 version in Zeppelin on HDP 25 sandbox

@Mikolaj Habdank-Wojewodzki Thats something weird.. lets go to basics again :-) Forget the newly created spark2 interpreter. Just Try editing the spark interpreter which came originally with Zeppelin. Change these properties

SPARK_HOME = /usr/hdp/current/spark2-client

SPARK_MAJOR_VERSION = 2

Save the settings and restart the interpreter.

Now, on the zeppelin notebook - try to run this

%spark

spark.sparkContext.version

and see if this works. I suspect the issue is with the new interpreter creation and not the configs which we are trying to do

Re: Is it possible to run Spark 2.0 version in Zeppelin on HDP 25 sandbox

New Contributor

Sadly this doesn't work also:

12636-prod-hadoop1-node1-local-9995-notebook-2canjuq7q.png

Re: Is it possible to run Spark 2.0 version in Zeppelin on HDP 25 sandbox

New Contributor

This is exactly the error I'm seeing with Spark 2, HDP 2.5, and Zeppelin.

Re: Is it possible to run Spark 2.0 version in Zeppelin on HDP 25 sandbox

New Contributor

Its not possible to run Spark2 with Zeppelin 0.6 which is the default in HDP 2.5 Sandbox.

Please find the detail discussion and bug below.

https://community.hortonworks.com/questions/59869/spark-2-and-zeppelin.html

The only way is the install Zeppelin 0.7 separately and link Spark and Iivy.

HTH

Amod

Highlighted

Re: Is it possible to run Spark 2.0 version in Zeppelin on HDP 25 sandbox

New Contributor

Hei can i use just spark2 only and delete spark1 for zeppelin?

Don't have an account?
Coming from Hortonworks? Activate your account here