Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Upgrade spark 1.5 to 1.6 HDP 2.3

avatar
Contributor

Hi all,

I just upgrade spark 1.5 to 1.6 on HDP 2.3

I'm following this tutorial here

and i got this error while try to run pysprak from terminal

/usr/bin/pyspark: line 22: /usr/bin/load-spark-env.sh: No such file or directory
/usr/bin/spark-class: line 23: /usr/bin/load-spark-env.sh: No such file or directory
ls: cannot access /usr/assembly/target/scala-: No such file or directory
Failed to find Spark assembly in /usr/assembly/target/scala-.
You need to build Spark before running this program.
1 ACCEPTED SOLUTION

avatar
Master Mentor

@cokorda putra susila make sure to install Spark client on the host you are running this from. Go to host components page and install Spark client.

View solution in original post

12 REPLIES 12

avatar
Master Mentor
@cokorda putra susila

rpm -qa | grep -i spark -> whats the output?

Did you run yum install spark_2_3_4_1_10-master -y ?

avatar
Contributor

hi @Neeraj Sabharwal

rpm -qa | grep -i spark -> this is the output

spark_2_3_4_0_3485-master-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-python-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-master-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-worker-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-python-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-1.5.2.2.3.4.0-3485.el6.noarch

Yes i run yum install spark_2_3_4_1_10-master -y as it the tutorial frow the link

avatar
Master Mentor

Main issue is because of setting SPARK_HOME

export SPARK_HOME=/usr/hdp/2.3.4.1-10/spark/

avatar
Master Mentor

@cokorda putra susila make sure to install Spark client on the host you are running this from. Go to host components page and install Spark client.

avatar
Contributor

I thinks i already install the spark client @Artem Ervits

I install spark 1.5 from ambari first, after that i upgrade the spark version from 1.5 to 1.6

When i try to run the pyspark i got this error.

It's possible to upgrade spark from latest HDP version?

avatar
Master Mentor

Wait for HDP 2.4 which will be released very soon and has full support for 1.6 Spark @cokorda putra susila

avatar
Contributor

Do you know when the estimation release for HDP 2.4? @Artem Ervits

avatar
Master Mentor

I don't, look for it within the next 4-8 weeks. @cokorda putra susila

avatar
Contributor

thanks for the information @Artem Ervits

For now i can run spark 1.6 from

/usr/hdp/2.3.4.1-10/spark/bin