Support Questions
Find answers, ask questions, and share your expertise

upgrading spark to 2.4

New Contributor

Dear All,

 

On one particular cluster we have spark 2.3 running(CDH 5.15.1). can it be possible to only upgrade spark to 2.4 ? by remaining on the same CDH version(5.15.1). will it compatible with existing service ?. kindly suggest  

1 ACCEPTED SOLUTION

Accepted Solutions

Re: upgrading spark to 2.4

Rising Star

Hi @javidshaik 

 

Yes based on cloudera documentation, it is not supported multiple versions under same Cloudera Manager Server.

View solution in original post

6 REPLIES 6

Re: upgrading spark to 2.4

Rising Star

Hi @javidshaik 

 

CDH 5.x and HDP 2.X version clusters has reached end of life support. Better upgrade your cluster to CDH 6.X/CDP 7.X version.

 

Both CDH 6.X and CDP 7.X clusters will support Spark 2.4 versions.

 

Please refer following documentation:

https://www.cloudera.com/legal/policies/support-lifecycle-policy.html

 

Re: upgrading spark to 2.4

New Contributor

Hi @RangaReddy

 

Yeah true i agree. thanks, but just wanted to know is it feasible or not to only update to spark2.4 on the cluster  ? or will there be any compatibility issues.  Appreciate your response 

Re: upgrading spark to 2.4

Rising Star

Hi @javidshaik 

 

I have checked with internal team. We can migrate Spark version from Spark 2.3 to 2.4 mentioned the details in below document.

 

2.4 Release 2CDH 5.10 and any higher CDH 5.x versions
2.4 Release 1CDH 5.10 and any higher CDH 5.x versions

 

https://docs.cloudera.com/documentation/spark2/latest/topics/spark2_requirements.html

 

But Spark 2.3 -> 2.4 version changes have higher potential of risks.

 

If you are satisfied with my answer please Accept as solution.

Re: upgrading spark to 2.4

New Contributor

hi @RangaReddy 

 

thanks for the below details.

 

Basically i have two(PROD & Stage) CDH (5.15.1) clusters with spark 2.3(2.3.0.cloudera4-1.cdh5.13.3.p0.611179) running on each cluster  under one CM. so i wanted to upgrade to spark2.4 only on one(stage) CDH(5.15.1) cluster. so as per below documentation link provided its being said

 

" All CDH clusters managed by the same Cloudera Manager Server must use exactly the same version of CDS Powered by Apache Spark. For example, you cannot use the built-in CDH Spark service, a CDS 2.1 service, and a CDS 2.2 service. You must choose only one CDS 2 Powered by Apache Spark release"

 

So i believe we cannot perform upgrade on this right. kindly confirm 

Re: upgrading spark to 2.4

Rising Star

Hi @javidshaik 

 

Yes based on cloudera documentation, it is not supported multiple versions under same Cloudera Manager Server.

View solution in original post

Re: upgrading spark to 2.4

New Contributor

ok thanks for the confirmation @RangaReddy