Issues upgrading Spark from Spark 1.3 -> Spark 1.4

[#1]

Hello,

I have Spark 1.2 running on CDH 5.3.1 and would like to upgrade to Spark 1.3

1) According to these two documentations for 5.4 and 5.0.x

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_spark_install.ht...

http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-0-0/CDH5-Installation-Guide/cdh5ig...

The command is: 

sudo yum install spark-core spark-master spark-worker spark-history-server spark-python

However, it is saying there is no spark-core, etc. avaialable 

2) Do I have to upgrade CDH to run Spark 1.3?

Thanks!

[#2 · Accepted solution]

These are instructions for installing via packages, which is not the usual way to do it. Do you really intend this?

If so have you set up the Cloudera repos?

Generally you manage using parcels, and yes updating Spark means updating CDH, since you're talking about updating many other harmonized dependencies.

[#3 · Accepted solution]

These are instructions for installing via packages, which is not the usual way to do it. Do you really intend this?

If so have you set up the Cloudera repos?

Generally you manage using parcels, and yes updating Spark means updating CDH, since you're talking about updating many other harmonized dependencies.

[#4]

Thank you Sean for the answer, I actually misspoke and just need to upgrade to Spark 1.3 (I'm using Spark 1.2). 

I've been trying to use this guide:

https://s3.amazonaws.com/quickstart-reference/cloudera/hadoop/latest/doc/Cloudera_EDH_on_AWS.pdf

But I am still only getting Spark 1.2, do you have any suggestions on how I can use this guide to get Spark 1.3?

[#5 · Accepted solution]

We do not support upgrading Spark without upgrading the rest of CDH.

Spark is compiled against a version of Hadoop and the versions of Hadoop can change between releases of CDH.

You also need to take into account the dependencies of Spark (like Hive) which might change between versions.

Even if you would be able to upgrade the package you might get weird failures due to the dependency breakage.

Wilfred