Hue Notebook how do I switch between spark versions I am using livy,Hue on HDP 2.5 Livy not switching spark version from 1.6 to 2.0

[#1]

Hi I am trying to run spark jobs using hue notebook but everything runs in spark 1.6 how can I switch or select spark 2. I am running on HDP 2.5. Hope its not a silly question and thanks for any help.

,

Hi I am trying to use hue notebook on HDP2.5 to run spark jobs and I want to run them in spark 2 but it seems to stay stuck in 1.6 I am not sure how to switch or specify that I would like to run my test code in spark 2. Hopefully this is not a silly question. Super thanks for any help.

[#2 · Accepted solution]

Livy 0.2 does not work with Spark 2 that I have found. Upgrading to Livy 0.3 and modifying the livy-env.sh to:

export SPARK_HOME=/usr/hdp/current/spark2-client

export SPARK_MAJOR_VERSION=2

This also works for Zeppelin 0.7. With livy 0.2 Zeppelin was throwing a 404 error attempting to get the version from livy.

[#3]

Hi jhorsch, thank you so much for taking the time to answer. You are an absolute superstar!!!! It worked a charm just perfect! Our Data Scientist says she owes you a beer!!!

Super thanks

Stephen