question Re: Steps to upgrade to Spark 1.5.1 on Sandbox in Archives of Support Questions (Read Only)

Steps to upgrade to Spark 1.5.1 on Sandbox

ssen — Wed, 14 Oct 2015 21:47:23 GMT

Has anyone tried manually upgrading to Spark 1.5.1 on Hortonworks Sandbox and faced any issues?

Re: Steps to upgrade to Spark 1.5.1 on Sandbox

abajwa — Thu, 15 Oct 2015 01:29:40 GMT

Did not upgrade the existing Spark on sandbox, but installed in a separate location while playing with Zeppelin and it worked fine. Below is the script I used to set it up (see readme for Ambari service for Zeppelin for more info)

sudo useradd zeppelin
sudo su zeppelin
cd /home/zeppelin
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.5.0-bin-hadoop2.6.tgz -O spark-1.5.0.tgz
tar -xzvf spark-1.5.0.tgz
export HDP_VER=`hdp-select status hadoop-client | sed 's/hadoop-client - \(.*\)/\1/'`
echo "spark.driver.extraJavaOptions -Dhdp.version=$HDP_VER" >> spark-1.5.0-bin-hadoop2.6/conf/spark-defaults.conf
echo "spark.yarn.am.extraJavaOptions -Dhdp.version=$HDP_VER" >> spark-1.5.0-bin-hadoop2.6/conf/spark-defaults.conf
exit

Re: Steps to upgrade to Spark 1.5.1 on Sandbox

nsabharwal — Thu, 15 Oct 2015 18:41:04 GMT

Try this . Thanks to @Randy Gelhausen

Re: Steps to upgrade to Spark 1.5.1 on Sandbox

gbraccialli3 — Tue, 17 Nov 2015 01:35:09 GMT

I used steps from our blog and it worked:

https://hortonworks.com/hadoop-tutorial/apache-spark-1-5-1-technical-preview-with-hdp-2-3/

Re: Steps to upgrade to Spark 1.5.1 on Sandbox

nsabharwal — Tue, 17 Nov 2015 02:13:28 GMT

@Guilherme Braccialli how about upgrade of existing spark install ?

Re: Steps to upgrade to Spark 1.5.1 on Sandbox

gbraccialli3 — Tue, 17 Nov 2015 02:46:36 GMT

@Neeraj should be the same, add hdp-updated repo, yum install, copy hive-site.xml

Re: Steps to upgrade to Spark 1.5.1 on Sandbox

awatson — Wed, 18 Nov 2015 03:02:03 GMT

@Saptak Sen Why try Spark 1.5.1 when you can try 1.5.2?

Spark is sort of like a client, so you can easily run multiple versions of Spark on one cluster. Simply download Spark from their the apache website - https://spark.apache.org/downloads.html - and invoke it to run it (can configure Zeppelin to use it or just run via the CLI).

Re: Steps to upgrade to Spark 1.5.1 on Sandbox

gbraccialli3 — Thu, 19 Nov 2015 08:25:10 GMT

@Andrew Watson I understood Hortonworks is going to support 1.5.1 in December and not 1.5.2, that would be the reason to use 1.5.1 instead of 1.5.2.