spark2 not accessable

[#1]

I upgraded my HDP2.4 to HDP2.5 and it installed spark2 successfully and I see it also as green and no errors in the HDP console but when on command line I see the version its showing me as 1.6.2 still ?

[root@hadoop5 ~]# spark-shell --version SPARK_MAJOR_VERSION is set to 2, using Spark2 Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.2 /_/

Type --help for more information. [root@hadoop5 ~]# echo $SPARK_MAJOR_VERSION 2

[#2 · Accepted solution]

@Sami Ahmad, not sure if this'll help, but did you set the env variable within the spark2 directory? (See http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_spark-component-guide/content/spark-choos...

[#3]

hi lgeorge if you look at my message I am showing the variable set , so this is not the issue

[#4]

also if you look into the message its saying "SPARK_MAJOR_VERSION is set to 2 "

[#5]

I even tried running the spark-submit from the spark2 directory , still shows me as version 1.6.2 ?

[root@hadoop1 ~]# echo $SPARK_MAJOR_VERSION 2 [root@hadoop1 ~]# /usr/hdp/2.5.0.0-1245/spark2/bin/spark-submit --version Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.2 /_/

Type --help for more information. [root@hadoop1 ~]#

[#6]

SPARK_MAJOR_VERSION was set but the SPARK_HOME wasn't .now its working .

thanks for your help

[#7]

@Sami Ahmad glad it's working! That's interesting. & thanks for clarifying; I wasn't sure if the EV was set in the spark2 directory (vs. spark1).