question Re: Upgrade spark 1.5 to 1.6 HDP 2.3 in Archives of Support Questions (Read Only)

Upgrade spark 1.5 to 1.6 HDP 2.3

cokorda_putra — Wed, 17 Feb 2016 20:11:47 GMT

Hi all,

I just upgrade spark 1.5 to 1.6 on HDP 2.3

I'm following this tutorial here

and i got this error while try to run pysprak from terminal

/usr/bin/pyspark: line 22: /usr/bin/load-spark-env.sh: No such file or directory
/usr/bin/spark-class: line 23: /usr/bin/load-spark-env.sh: No such file or directory
ls: cannot access /usr/assembly/target/scala-: No such file or directory
Failed to find Spark assembly in /usr/assembly/target/scala-.
You need to build Spark before running this program.

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

nsabharwal — Wed, 17 Feb 2016 20:19:29 GMT

@cokorda putra susila

rpm -qa | grep -i spark -> whats the output?

Did you run yum install spark_2_3_4_1_10-master -y ?

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

aervits — Wed, 17 Feb 2016 20:36:36 GMT

@cokorda putra susila make sure to install Spark client on the host you are running this from. Go to host components page and install Spark client.

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

cokorda_putra — Wed, 17 Feb 2016 20:37:39 GMT

hi @Neeraj Sabharwal

rpm -qa | grep -i spark -> this is the output

spark_2_3_4_0_3485-master-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-python-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-master-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-worker-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-python-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-1.5.2.2.3.4.0-3485.el6.noarch

Yes i run yum install spark_2_3_4_1_10-master -y as it the tutorial frow the link

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

cokorda_putra — Wed, 17 Feb 2016 20:52:35 GMT

I thinks i already install the spark client @Artem Ervits

I install spark 1.5 from ambari first, after that i upgrade the spark version from 1.5 to 1.6

When i try to run the pyspark i got this error.

It's possible to upgrade spark from latest HDP version?

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

aervits — Wed, 17 Feb 2016 20:55:26 GMT

Wait for HDP 2.4 which will be released very soon and has full support for 1.6 Spark @cokorda putra susila

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

cokorda_putra — Wed, 17 Feb 2016 21:01:38 GMT

Do you know when the estimation release for HDP 2.4? @Artem Ervits

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

aervits — Wed, 17 Feb 2016 21:02:59 GMT

I don't, look for it within the next 4-8 weeks. @cokorda putra susila

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

cokorda_putra — Wed, 17 Feb 2016 22:09:23 GMT

thanks for the information @Artem Ervits

For now i can run spark 1.6 from

/usr/hdp/2.3.4.1-10/spark/bin

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

aervits — Wed, 17 Feb 2016 22:10:46 GMT

Great, @cokorda putra susila please accept one of the answers as best practice.

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

nsabharwal — Wed, 17 Feb 2016 22:12:57 GMT

Main issue is because of setting SPARK_HOME

export SPARK_HOME=/usr/hdp/2.3.4.1-10/spark/

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

vshukla — Thu, 18 Feb 2016 02:53:59 GMT

Officially upgrade from Spark 1.5.2 > 1.6 is part of HDP 2.4 that is coming out March 1st.

The Spark 1.6 tutorial is for evaluating Spark 1.6 and not for evaluation of upgrade instructions.

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

nsabharwal — Thu, 18 Feb 2016 03:05:26 GMT

@cokorda putra susila