Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

upgrading spark to 2.4

avatar
Explorer

Dear All,

 

On one particular cluster we have spark 2.3 running(CDH 5.15.1). can it be possible to only upgrade spark to 2.4 ? by remaining on the same CDH version(5.15.1). will it compatible with existing service ?. kindly suggest  

1 ACCEPTED SOLUTION

avatar
Super Collaborator

Hi @javidshaik 

 

Yes based on cloudera documentation, it is not supported multiple versions under same Cloudera Manager Server.

View solution in original post

12 REPLIES 12

avatar
Super Collaborator

Hi @javidshaik 

 

CDH 5.x and HDP 2.X version clusters has reached end of life support. Better upgrade your cluster to CDH 6.X/CDP 7.X version.

 

Both CDH 6.X and CDP 7.X clusters will support Spark 2.4 versions.

 

Please refer following documentation:

https://www.cloudera.com/legal/policies/support-lifecycle-policy.html

 

avatar
Explorer

Hi @RangaReddy

 

Yeah true i agree. thanks, but just wanted to know is it feasible or not to only update to spark2.4 on the cluster  ? or will there be any compatibility issues.  Appreciate your response 

avatar
Super Collaborator

Hi @javidshaik 

 

I have checked with internal team. We can migrate Spark version from Spark 2.3 to 2.4 mentioned the details in below document.

 

2.4 Release 2CDH 5.10 and any higher CDH 5.x versions
2.4 Release 1CDH 5.10 and any higher CDH 5.x versions

 

https://docs.cloudera.com/documentation/spark2/latest/topics/spark2_requirements.html

 

But Spark 2.3 -> 2.4 version changes have higher potential of risks.

 

If you are satisfied with my answer please Accept as solution.

avatar
Explorer

hi @RangaReddy 

 

thanks for the below details.

 

Basically i have two(PROD & Stage) CDH (5.15.1) clusters with spark 2.3(2.3.0.cloudera4-1.cdh5.13.3.p0.611179) running on each cluster  under one CM. so i wanted to upgrade to spark2.4 only on one(stage) CDH(5.15.1) cluster. so as per below documentation link provided its being said

 

" All CDH clusters managed by the same Cloudera Manager Server must use exactly the same version of CDS Powered by Apache Spark. For example, you cannot use the built-in CDH Spark service, a CDS 2.1 service, and a CDS 2.2 service. You must choose only one CDS 2 Powered by Apache Spark release"

 

So i believe we cannot perform upgrade on this right. kindly confirm 

avatar
Super Collaborator

Hi @javidshaik 

 

Yes based on cloudera documentation, it is not supported multiple versions under same Cloudera Manager Server.

avatar
Explorer

ok thanks for the confirmation @RangaReddy 

avatar

Hi 

 

As mentioned below, Spark 2.3 -> 2.4 version changes have higher potential of risks.

May i know what are the those risks. It would be more helpful if you share right link for these and also share what the considerations need to be taken while upgrading form spark 1.6 to 2.4 and spark 2.3 to spark 2.4

 

Thanks

Rajesh

avatar
Super Collaborator

avatar

Thanks for the update.

Is there any changes in sizing and memory for spark 2.4?