Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark 2.2 and Livy

Solved Go to solution

Spark 2.2 and Livy

Expert Contributor

Hello,

 

I have both Spark 1.6 and Spark 2.2 installed in my cluster through CDH. Normally my Livy server starts with default Spark 1.6 but now I want to start my Livy with Spark 2.2 which I figured maybe by changing the SPARK_HOME to point to SPARK2 would do the trick:

 

Previously:

export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark

export SPARK_CONF_DIR=$SPARK_HOME/conf

 

New:

export SPARK_HOME=/opt/cloudera/parcels/SPARK2

export SPARK_CONF_DIR=$SPARK_HOME/meta

 

But this will led to an error which is understandable as Spark2 in Cloudera is spark2-submit:

 

Exception in thread "main" java.io.IOException: Cannot run program "/opt/cloudera/parcels/SPARK2/bin/spark-submit": error=2, No such file or directory


Is there anyway to configure the Livy to find the right spark2-submit and not the default name? I looked every where in the config and the code but maybe I missed something.

 

Many thanks,

Maziyar

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Spark 2.2 and Livy

Champion
I haven't done this yet but it should do the trick. You need to update the alternatives to make Spark2 the default. This will make it the default across the board and not just for Livy. So make sure you are ready for that.

https://www.cloudera.com/documentation/spark2/latest/topics/spark2_admin.html
4 REPLIES 4

Re: Spark 2.2 and Livy

Champion
I haven't done this yet but it should do the trick. You need to update the alternatives to make Spark2 the default. This will make it the default across the board and not just for Livy. So make sure you are ready for that.

https://www.cloudera.com/documentation/spark2/latest/topics/spark2_admin.html

Re: Spark 2.2 and Livy

Expert Contributor

It worked! I just ran the script on the node that is my Livy server. 

 

Thank you :)

Highlighted

Re: Spark 2.2 and Livy

New Contributor

Is there a way of doing this without having to make Spark2 the default? I tried to search among Livy conf parameters, but I couldn't find anything.

Re: Spark 2.2 and Livy

Expert Contributor
The Spark 2 now is the only Spark that is supported by CDH 6.x so I am not sure you will get any reply here.
Is there any reason you are still in Spark 1.6.x?
Don't have an account?
Coming from Hortonworks? Activate your account here