question Re: Issues upgrading Spark from Spark 1.3 -> Spark 1.4 in Archives of Support Questions (Read Only)

Issues upgrading Spark from Spark 1.3 -> Spark 1.4

ClouderaUser52 — Fri, 16 Sep 2022 09:32:29 GMT

Hello,

I have Spark 1.2 running on CDH 5.3.1 and would like to upgrade to Spark 1.3

1) According to these two documentations for 5.4 and 5.0.x

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_spark_install.html

http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-0-0/CDH5-Installation-Guide/cdh5ig_spark_install.html

The command is:

sudo yum install spark-core spark-master spark-worker spark-history-server spark-python

However, it is saying there is no spark-core, etc. avaialable

2) Do I have to upgrade CDH to run Spark 1.3?

Thanks!

Re: Issues upgrading Spark from Spark 1.3 -> Spark 1.4

srowen — Fri, 26 Jun 2015 06:36:19 GMT

These are instructions for installing via packages, which is not the usual way to do it. Do you really intend this?

If so have you set up the Cloudera repos?

Generally you manage using parcels, and yes updating Spark means updating CDH, since you're talking about updating many other harmonized dependencies.

Re: Issues upgrading Spark from Spark 1.3 -> Spark 1.4

Wilfred — Mon, 29 Jun 2015 07:43:30 GMT

We do not support upgrading Spark without upgrading the rest of CDH.

Spark is compiled against a version of Hadoop and the versions of Hadoop can change between releases of CDH.

You also need to take into account the dependencies of Spark (like Hive) which might change between versions.

Even if you would be able to upgrade the package you might get weird failures due to the dependency breakage.

Wilfred

Re: Issues upgrading Spark from Spark 1.3 -> Spark 1.4

ClouderaUser52 — Tue, 30 Jun 2015 02:02:57 GMT

Thank you Sean for the answer, I actually misspoke and just need to upgrade to Spark 1.3 (I'm using Spark 1.2).

I've been trying to use this guide:

https://s3.amazonaws.com/quickstart-reference/cloudera/hadoop/latest/doc/Cloudera_EDH_on_AWS.pdf

But I am still only getting Spark 1.2, do you have any suggestions on how I can use this guide to get Spark 1.3?