Upgrade spark 1.5 to 1.6 HDP 2.3

[#1]

Hi all,

I just upgrade spark 1.5 to 1.6 on HDP 2.3

I'm following this tutorial here

and i got this error while try to run pysprak from terminal

/usr/bin/pyspark: line 22: /usr/bin/load-spark-env.sh: No such file or directory
/usr/bin/spark-class: line 23: /usr/bin/load-spark-env.sh: No such file or directory
ls: cannot access /usr/assembly/target/scala-: No such file or directory
Failed to find Spark assembly in /usr/assembly/target/scala-.
You need to build Spark before running this program.

[#2]

@cokorda putra susila

rpm -qa | grep -i spark -> whats the output?

Did you run yum install spark_2_3_4_1_10-master -y ?

[#3]

hi @Neeraj Sabharwal

rpm -qa | grep -i spark -> this is the output

spark_2_3_4_0_3485-master-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-python-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-master-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-worker-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-python-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-1.5.2.2.3.4.0-3485.el6.noarch

Yes i run yum install spark_2_3_4_1_10-master -y as it the tutorial frow the link

[#4]

Main issue is because of setting SPARK_HOME

export SPARK_HOME=/usr/hdp/2.3.4.1-10/spark/

[#5 · Accepted solution]

@cokorda putra susila make sure to install Spark client on the host you are running this from. Go to host components page and install Spark client.

[#6]

I thinks i already install the spark client @Artem Ervits

I install spark 1.5 from ambari first, after that i upgrade the spark version from 1.5 to 1.6

When i try to run the pyspark i got this error.

It's possible to upgrade spark from latest HDP version?

[#7]

Wait for HDP 2.4 which will be released very soon and has full support for 1.6 Spark @cokorda putra susila

[#8]

Do you know when the estimation release for HDP 2.4? @Artem Ervits

[#9]

I don't, look for it within the next 4-8 weeks. @cokorda putra susila

[#10]

thanks for the information @Artem Ervits

For now i can run spark 1.6 from

/usr/hdp/2.3.4.1-10/spark/bin

[#11]

Great, @cokorda putra susila please accept one of the answers as best practice.

[#12]

Officially upgrade from Spark 1.5.2 > 1.6 is part of HDP 2.4 that is coming out March 1st.

The Spark 1.6 tutorial is for evaluating Spark 1.6 and not for evaluation of upgrade instructions.

[#13]

@cokorda putra susila