In Zeppelin, when I type: sc.version it comes back 1.6.2.
But I would like to run Spark 2.0 in Zeppelin instead. Is there a way to set this up??
I've restarted zeppelin, recreated notebook, verified that interpreter is enabled (by default it's always disabled) but have still same exception. Please, can you verify that below configuration is correct:
Can you also post your %spark config snapshot side-by-side? Also Can you give this thing a try - using %spark2.spark instead of %spark2 - like this
There are some more screens:
There are two things which looks interesting:
1) the output for interpreter spark2 looks different than it two other cases
2) spark2 is always by default off when I create new notebook. then I'm enabling it after I create notebook.
@Mikolaj Habdank-Wojewodzki Thats something weird.. lets go to basics again 🙂 Forget the newly created spark2 interpreter. Just Try editing the spark interpreter which came originally with Zeppelin. Change these properties
SPARK_HOME = /usr/hdp/current/spark2-client
SPARK_MAJOR_VERSION = 2
Save the settings and restart the interpreter.
Now, on the zeppelin notebook - try to run this
and see if this works. I suspect the issue is with the new interpreter creation and not the configs which we are trying to do
Its not possible to run Spark2 with Zeppelin 0.6 which is the default in HDP 2.5 Sandbox.
Please find the detail discussion and bug below.
The only way is the install Zeppelin 0.7 separately and link Spark and Iivy.