Support Questions

Find answers, ask questions, and share your expertise

Spark2 not working

avatar
New Contributor

I recently installed HDP 2.5 on a 3 node cluster using Ambari. I have installed Spark 2.0 along with it and have set SPARK_MAJOR_VERSION to 2. The spark-shell starts fine for 1.6.2 but throws the following error for Spark 2.0.

spark-shell 
SPARK_MAJOR_VERSION is set to 2, using Spark2
/usr/hdp/current/spark2-client/bin/spark-shell: line 57:
/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/spark-2.0.0-bin-hadoop2.6/bin/spark-submit: No such file or directory

Can someone help with this error?

1 ACCEPTED SOLUTION

avatar
New Contributor

Just realized SPARK_HOME wasn't set properly...SPARK_HOME was set to the old cloudera version we had on that cluster...resetting it made it work

View solution in original post

2 REPLIES 2

avatar

"/opt/cloudera/parcels/CDH-5.7..." doesn't sound like HDP 2.5. Do you have a conflicting install of CDH and HDP?

avatar
New Contributor

Just realized SPARK_HOME wasn't set properly...SPARK_HOME was set to the old cloudera version we had on that cluster...resetting it made it work