question Spark2 not working in Archives of Support Questions (Read Only)

Spark2 not working

vioravis — Mon, 07 Nov 2016 20:17:36 GMT

I recently installed HDP 2.5 on a 3 node cluster using Ambari. I have installed Spark 2.0 along with it and have set SPARK_MAJOR_VERSION to 2. The spark-shell starts fine for 1.6.2 but throws the following error for Spark 2.0.

spark-shell 
SPARK_MAJOR_VERSION is set to 2, using Spark2
/usr/hdp/current/spark2-client/bin/spark-shell: line 57:
/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/spark-2.0.0-bin-hadoop2.6/bin/spark-submit: No such file or directory

Can someone help with this error?

Re: Spark2 not working

bwalter1 — Mon, 07 Nov 2016 20:30:56 GMT

"/opt/cloudera/parcels/CDH-5.7..." doesn't sound like HDP 2.5. Do you have a conflicting install of CDH and HDP?

Re: Spark2 not working

vioravis — Mon, 07 Nov 2016 20:34:13 GMT

Just realized SPARK_HOME wasn't set properly...SPARK_HOME was set to the old cloudera version we had on that cluster...resetting it made it work