upgrading spark to 2.4

12 replies
9823 views
Translate All

Dear All,

On one particular cluster we have spark 2.3 running(CDH 5.15.1). can it be possible to only upgrade spark to 2.4 ? by remaining on the same CDH version(5.15.1). will it compatible with existing service ?. kindly suggest

Apache Spark

RangaReddyAnswer

Gold ⭐️

Hi @javidshaik

Yes based on cloudera documentation, it is not supported multiple versions under same Cloudera Manager Server.

RangaReddy

Gold ⭐️

Hi @javidshaik

CDH 5.x and HDP 2.X version clusters has reached end of life support. Better upgrade your cluster to CDH 6.X/CDP 7.X version.

Both CDH 6.X and CDP 7.X clusters will support Spark 2.4 versions.

Please refer following documentation:

https://www.cloudera.com/legal/policies/support-lifecycle-policy.html

Translate

javidshaikAuthor

Silver ⭐️⭐️

Hi @RangaReddy

Yeah true i agree. thanks, but just wanted to know is it feasible or not to only update to spark2.4 on the cluster ? or will there be any compatibility issues. Appreciate your response

Translate

RangaReddy

Gold ⭐️

Hi @javidshaik

I have checked with internal team. We can migrate Spark version from Spark 2.3 to 2.4 mentioned the details in below document.

2.4 Release 2	CDH 5.10 and any higher CDH 5.x versions
2.4 Release 1	CDH 5.10 and any higher CDH 5.x versions

https://docs.cloudera.com/documentation/spark2/latest/topics/spark2_requirements.html

But Spark 2.3 -> 2.4 version changes have higher potential of risks.

If you are satisfied with my answer please Accept as solution.

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.