question Re: spark2 not accessable in Support Questions

spark2 not accessable

aliyesami — Thu, 29 Sep 2016 21:59:11 GMT

I upgraded my HDP2.4 to HDP2.5 and it installed spark2 successfully and I see it also as green and no errors in the HDP console but when on command line I see the version its showing me as 1.6.2 still ?

[root@hadoop5 ~]# spark-shell --version SPARK_MAJOR_VERSION is set to 2, using Spark2 Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.2 /_/

Type --help for more information. [root@hadoop5 ~]# echo $SPARK_MAJOR_VERSION 2

Re: spark2 not accessable

lgeorge — Thu, 29 Sep 2016 22:17:20 GMT

@Sami Ahmad, not sure if this'll help, but did you set the env variable within the spark2 directory? (See http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_spark-component-guide/content/spark-choose-version.html.)

Re: spark2 not accessable

aliyesami — Thu, 29 Sep 2016 22:30:49 GMT

hi lgeorge if you look at my message I am showing the variable set , so this is not the issue

Re: spark2 not accessable

aliyesami — Thu, 29 Sep 2016 22:32:18 GMT

also if you look into the message its saying "SPARK_MAJOR_VERSION is set to 2 "

Re: spark2 not accessable

aliyesami — Thu, 29 Sep 2016 23:20:39 GMT

I even tried running the spark-submit from the spark2 directory , still shows me as version 1.6.2 ?

[root@hadoop1 ~]# echo $SPARK_MAJOR_VERSION 2 [root@hadoop1 ~]# /usr/hdp/2.5.0.0-1245/spark2/bin/spark-submit --version Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.2 /_/

Type --help for more information. [root@hadoop1 ~]#

Re: spark2 not accessable

aliyesami — Thu, 29 Sep 2016 23:27:40 GMT

SPARK_MAJOR_VERSION was set but the SPARK_HOME wasn't .now its working .

thanks for your help

Re: spark2 not accessable

lgeorge — Fri, 30 Sep 2016 01:20:00 GMT

@Sami Ahmad glad it's working! That's interesting. & thanks for clarifying; I wasn't sure if the EV was set in the spark2 directory (vs. spark1).