Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Spark upgrade

Spark upgrade

New Contributor

I am using CDH 5.4.8. and Spark version is 1.3.0. I would like to upgrade Spark version to either 1.5 or 1.6. 

Do I need to do full CDH upgrade for this spark upgrade?

If not, willl the spark upgrade affect any other layers of CDH?

1 REPLY 1

Re: Spark upgrade

Master Collaborator

I think it's certainly easiest to update CDH, because you get other updates and get a harmonized distribution. CDH 5.7 includes the latest Spark 1.6. The only downside, I suppose, is that you are updating all components and there is a non-zero problem of a problem or incompatibility, but, you'll almost surely gain much more than you risk.

 

In any event, you can always build and run any version of Spark, and it should Just Work in the main. It does require a bit of understanding of how to build and execute a Spark distribution.