Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Upgrade spark 1.5 to 1.6 HDP 2.3

Solved Go to solution
Highlighted

Upgrade spark 1.5 to 1.6 HDP 2.3

Hi all,

I just upgrade spark 1.5 to 1.6 on HDP 2.3

I'm following this tutorial here

and i got this error while try to run pysprak from terminal

/usr/bin/pyspark: line 22: /usr/bin/load-spark-env.sh: No such file or directory
/usr/bin/spark-class: line 23: /usr/bin/load-spark-env.sh: No such file or directory
ls: cannot access /usr/assembly/target/scala-: No such file or directory
Failed to find Spark assembly in /usr/assembly/target/scala-.
You need to build Spark before running this program.
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

Mentor

@cokorda putra susila make sure to install Spark client on the host you are running this from. Go to host components page and install Spark client.

View solution in original post

12 REPLIES 12
Highlighted

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

@cokorda putra susila

rpm -qa | grep -i spark -> whats the output?

Did you run yum install spark_2_3_4_1_10-master -y ?

Highlighted

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

hi @Neeraj Sabharwal

rpm -qa | grep -i spark -> this is the output

spark_2_3_4_0_3485-master-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-python-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-master-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-worker-1.5.2.2.3.4.0-3485.el6.noarch
spark_2_3_4_1_10-python-1.6.0.2.3.4.1-10.el6.noarch
spark_2_3_4_0_3485-1.5.2.2.3.4.0-3485.el6.noarch

Yes i run yum install spark_2_3_4_1_10-master -y as it the tutorial frow the link

Highlighted

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

Main issue is because of setting SPARK_HOME

export SPARK_HOME=/usr/hdp/2.3.4.1-10/spark/
Highlighted

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

Mentor

@cokorda putra susila make sure to install Spark client on the host you are running this from. Go to host components page and install Spark client.

View solution in original post

Highlighted

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

I thinks i already install the spark client @Artem Ervits

I install spark 1.5 from ambari first, after that i upgrade the spark version from 1.5 to 1.6

When i try to run the pyspark i got this error.

It's possible to upgrade spark from latest HDP version?

Highlighted

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

Mentor

Wait for HDP 2.4 which will be released very soon and has full support for 1.6 Spark @cokorda putra susila

Highlighted

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

Do you know when the estimation release for HDP 2.4? @Artem Ervits

Highlighted

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

Mentor

I don't, look for it within the next 4-8 weeks. @cokorda putra susila

Highlighted

Re: Upgrade spark 1.5 to 1.6 HDP 2.3

thanks for the information @Artem Ervits

For now i can run spark 1.6 from

/usr/hdp/2.3.4.1-10/spark/bin

Don't have an account?
Coming from Hortonworks? Activate your account here