Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Upgrade spark 1.5 to 1.6 HDP 2.3

avatar
Contributor

Hi all,

I just upgrade spark 1.5 to 1.6 on HDP 2.3

I'm following this tutorial here

and i got this error while try to run pysprak from terminal

/usr/bin/pyspark: line 22: /usr/bin/load-spark-env.sh: No such file or directory
/usr/bin/spark-class: line 23: /usr/bin/load-spark-env.sh: No such file or directory
ls: cannot access /usr/assembly/target/scala-: No such file or directory
Failed to find Spark assembly in /usr/assembly/target/scala-.
You need to build Spark before running this program.
1 ACCEPTED SOLUTION

avatar
Master Mentor

@cokorda putra susila make sure to install Spark client on the host you are running this from. Go to host components page and install Spark client.

View solution in original post

12 REPLIES 12

avatar
Master Mentor

Great, @cokorda putra susila please accept one of the answers as best practice.

avatar

Officially upgrade from Spark 1.5.2 > 1.6 is part of HDP 2.4 that is coming out March 1st.

The Spark 1.6 tutorial is for evaluating Spark 1.6 and not for evaluation of upgrade instructions.

avatar
Master Mentor