Created 09-29-2016 02:59 PM
I upgraded my HDP2.4 to HDP2.5 and it installed spark2 successfully and I see it also as green and no errors in the HDP console but when on command line I see the version its showing me as 1.6.2 still ?
[root@hadoop5 ~]# spark-shell --version SPARK_MAJOR_VERSION is set to 2, using Spark2 Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.2 /_/
Type --help for more information. [root@hadoop5 ~]# echo $SPARK_MAJOR_VERSION 2
Created 09-29-2016 03:17 PM
Want to get a detailed solution you have to login/registered on the community
Register/LoginCreated 09-29-2016 03:17 PM
Want to get a detailed solution you have to login/registered on the community
Register/LoginCreated 09-29-2016 03:30 PM
hi lgeorge if you look at my message I am showing the variable set , so this is not the issue
Created 09-29-2016 03:32 PM
also if you look into the message its saying "SPARK_MAJOR_VERSION is set to 2 "
Created 09-29-2016 04:20 PM
I even tried running the spark-submit from the spark2 directory , still shows me as version 1.6.2 ?
[root@hadoop1 ~]# echo $SPARK_MAJOR_VERSION 2 [root@hadoop1 ~]# /usr/hdp/2.5.0.0-1245/spark2/bin/spark-submit --version Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.2 /_/
Type --help for more information. [root@hadoop1 ~]#
Created 09-29-2016 04:27 PM
SPARK_MAJOR_VERSION was set but the SPARK_HOME wasn't .now its working .
thanks for your help
Created 09-29-2016 06:20 PM
@Sami Ahmad glad it's working! That's interesting. & thanks for clarifying; I wasn't sure if the EV was set in the spark2 directory (vs. spark1).