upgrading spark to 2.4

[#1]

Dear All,

On one particular cluster we have spark 2.3 running(CDH 5.15.1). can it be possible to only upgrade spark to 2.4 ? by remaining on the same CDH version(5.15.1). will it compatible with existing service ?. kindly suggest  

[#2]

Hi @javidshaik 

CDH 5.x and HDP 2.X version clusters has reached end of life support. Better upgrade your cluster to CDH 6.X/CDP 7.X version.

Both CDH 6.X and CDP 7.X clusters will support Spark 2.4 versions.

Please refer following documentation:

https://www.cloudera.com/legal/policies/support-lifecycle-policy.html

[#3]

Hi @RangaReddy

Yeah true i agree. thanks, but just wanted to know is it feasible or not to only update to spark2.4 on the cluster  ? or will there be any compatibility issues.  Appreciate your response 

[#4]

Hi @javidshaik 

I have checked with internal team. We can migrate Spark version from Spark 2.3 to 2.4 mentioned the details in below document.

2.4 Release 2	CDH 5.10 and any higher CDH 5.x versions
2.4 Release 1	CDH 5.10 and any higher CDH 5.x versions

https://docs.cloudera.com/documentation/spark2/latest/topics/spark2_requirements.html

But Spark 2.3 -> 2.4 version changes have higher potential of risks.

If you are satisfied with my answer please Accept as solution.

[#5]

hi @RangaReddy 

thanks for the below details.

Basically i have two(PROD & Stage) CDH (5.15.1) clusters with spark 2.3(2.3.0.cloudera4-1.cdh5.13.3.p0.611179) running on each cluster  under one CM. so i wanted to upgrade to spark2.4 only on one(stage) CDH(5.15.1) cluster. so as per below documentation link provided its being said

" All CDH clusters managed by the same Cloudera Manager Server must use exactly the same version of CDS Powered by Apache Spark. For example, you cannot use the built-in CDH Spark service, a CDS 2.1 service, and a CDS 2.2 service. You must choose only one CDS 2 Powered by Apache Spark release"

So i believe we cannot perform upgrade on this right. kindly confirm 

[#6 · Accepted solution]

Hi @javidshaik 

Yes based on cloudera documentation, it is not supported multiple versions under same Cloudera Manager Server.

[#7]

ok thanks for the confirmation @RangaReddy 

[#8]

Hi 

As mentioned below, Spark 2.3 -> 2.4 version changes have higher potential of risks.

May i know what are the those risks. It would be more helpful if you share right link for these and also share what the considerations need to be taken while upgrading form spark 1.6 to 2.4 and spark 2.3 to spark 2.4

Thanks

Rajesh

[#9]

Hi @Rajeshhadoop 

Please find the few of the references:

1. https://spark.apache.org/docs/2.4.0/sql-migration-guide.html
2. https://blog.knoldus.com/migration-from-spark-1-x-to-spark-2-x/
3. https://docs.cloudera.com/documentation/enterprise/upgrade/topics/ug_spark_post.html

Note: We don't have document for upgrading straight from spark 1.x to spark 2.4

[#10]

Thanks for the update.

Is there any changes in sizing and memory for spark 2.4?