Reply
New Contributor
Posts: 1
Registered: ‎05-26-2016

Spark upgrade

I am using CDH 5.4.8. and Spark version is 1.3.0. I would like to upgrade Spark version to either 1.5 or 1.6. 

Do I need to do full CDH upgrade for this spark upgrade?

If not, willl the spark upgrade affect any other layers of CDH?

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Spark upgrade

I think it's certainly easiest to update CDH, because you get other updates and get a harmonized distribution. CDH 5.7 includes the latest Spark 1.6. The only downside, I suppose, is that you are updating all components and there is a non-zero problem of a problem or incompatibility, but, you'll almost surely gain much more than you risk.

 

In any event, you can always build and run any version of Spark, and it should Just Work in the main. It does require a bit of understanding of how to build and execute a Spark distribution.