On one particular cluster we have spark 2.3 running(CDH 5.15.1). can it be possible to only upgrade spark to 2.4 ? by remaining on the same CDH version(5.15.1). will it compatible with existing service ?. kindly suggest
CDH 5.x and HDP 2.X version clusters has reached end of life support. Better upgrade your cluster to CDH 6.X/CDP 7.X version.
Both CDH 6.X and CDP 7.X clusters will support Spark 2.4 versions.
Please refer following documentation:
I have checked with internal team. We can migrate Spark version from Spark 2.3 to 2.4 mentioned the details in below document.
|2.4 Release 2||CDH 5.10 and any higher CDH 5.x versions|
|2.4 Release 1||CDH 5.10 and any higher CDH 5.x versions|
But Spark 2.3 -> 2.4 version changes have higher potential of risks.
If you are satisfied with my answer please Accept as solution.
thanks for the below details.
Basically i have two(PROD & Stage) CDH (5.15.1) clusters with spark 2.3(2.3.0.cloudera4-1.cdh5.13.3.p0.611179) running on each cluster under one CM. so i wanted to upgrade to spark2.4 only on one(stage) CDH(5.15.1) cluster. so as per below documentation link provided its being said
" All CDH clusters managed by the same Cloudera Manager Server must use exactly the same version of CDS Powered by Apache Spark. For example, you cannot use the built-in CDH Spark service, a CDS 2.1 service, and a CDS 2.2 service. You must choose only one CDS 2 Powered by Apache Spark release"
So i believe we cannot perform upgrade on this right. kindly confirm