Created 11-07-2016 12:17 PM
I recently installed HDP 2.5 on a 3 node cluster using Ambari. I have installed Spark 2.0 along with it and have set SPARK_MAJOR_VERSION to 2. The spark-shell starts fine for 1.6.2 but throws the following error for Spark 2.0.
spark-shell SPARK_MAJOR_VERSION is set to 2, using Spark2 /usr/hdp/current/spark2-client/bin/spark-shell: line 57: /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/spark-2.0.0-bin-hadoop2.6/bin/spark-submit: No such file or directory
Can someone help with this error?
Created 11-07-2016 12:34 PM
Just realized SPARK_HOME wasn't set properly...SPARK_HOME was set to the old cloudera version we had on that cluster...resetting it made it work
Created 11-07-2016 12:30 PM
"/opt/cloudera/parcels/CDH-5.7..." doesn't sound like HDP 2.5. Do you have a conflicting install of CDH and HDP?
Created 11-07-2016 12:34 PM
Just realized SPARK_HOME wasn't set properly...SPARK_HOME was set to the old cloudera version we had on that cluster...resetting it made it work