question Re: upgrading spark to 2.4 in Support Questions

upgrading spark to 2.4

javidshaik — Sun, 27 Jun 2021 04:47:36 GMT

Dear All,

On one particular cluster we have spark 2.3 running(CDH 5.15.1). can it be possible to only upgrade spark to 2.4 ? by remaining on the same CDH version(5.15.1). will it compatible with existing service ?. kindly suggest

Re: upgrading spark to 2.4

RangaReddy — Mon, 28 Jun 2021 09:39:51 GMT

Hi @javidshaik

CDH 5.x and HDP 2.X version clusters has reached end of life support. Better upgrade your cluster to CDH 6.X/CDP 7.X version.

Both CDH 6.X and CDP 7.X clusters will support Spark 2.4 versions.

Please refer following documentation:

https://www.cloudera.com/legal/policies/support-lifecycle-policy.html

Re: upgrading spark to 2.4

javidshaik — Mon, 28 Jun 2021 09:43:27 GMT

Hi @RangaReddy

Yeah true i agree. thanks, but just wanted to know is it feasible or not to only update to spark2.4 on the cluster ? or will there be any compatibility issues. Appreciate your response

Re: upgrading spark to 2.4

RangaReddy — Mon, 28 Jun 2021 11:04:22 GMT

Hi @javidshaik

I have checked with internal team. We can migrate Spark version from Spark 2.3 to 2.4 mentioned the details in below document.

2.4 Release 2	CDH 5.10 and any higher CDH 5.x versions
2.4 Release 1	CDH 5.10 and any higher CDH 5.x versions

https://docs.cloudera.com/documentation/spark2/latest/topics/spark2_requirements.html

But Spark 2.3 -> 2.4 version changes have higher potential of risks.

If you are satisfied with my answer please Accept as solution.

Re: upgrading spark to 2.4

javidshaik — Mon, 28 Jun 2021 15:56:33 GMT

hi @RangaReddy

thanks for the below details.

Basically i have two(PROD & Stage) CDH (5.15.1) clusters with spark 2.3(2.3.0.cloudera4-1.cdh5.13.3.p0.611179) running on each cluster under one CM. so i wanted to upgrade to spark2.4 only on one(stage) CDH(5.15.1) cluster. so as per below documentation link provided its being said

" All CDH clusters managed by the same Cloudera Manager Server must use exactly the same version of CDS Powered by Apache Spark. For example, you cannot use the built-in CDH Spark service, a CDS 2.1 service, and a CDS 2.2 service. You must choose only one CDS 2 Powered by Apache Spark release"

So i believe we cannot perform upgrade on this right. kindly confirm

Re: upgrading spark to 2.4

RangaReddy — Tue, 29 Jun 2021 02:22:48 GMT

Hi @javidshaik

Yes based on cloudera documentation, it is not supported multiple versions under same Cloudera Manager Server.

Re: upgrading spark to 2.4

javidshaik — Tue, 29 Jun 2021 04:15:07 GMT

ok thanks for the confirmation @RangaReddy

Re: upgrading spark to 2.4

Rajeshhadoop — Fri, 18 Feb 2022 13:17:45 GMT

As mentioned below, Spark 2.3 -> 2.4 version changes have higher potential of risks.

May i know what are the those risks. It would be more helpful if you share right link for these and also share what the considerations need to be taken while upgrading form spark 1.6 to 2.4 and spark 2.3 to spark 2.4

Thanks

Rajesh

Re: upgrading spark to 2.4

RangaReddy — Mon, 21 Feb 2022 10:16:57 GMT

Hi @Rajeshhadoop

Please find the few of the references:

Note: We don't have document for upgrading straight from spark 1.x to spark 2.4

Re: upgrading spark to 2.4

Rajeshhadoop — Mon, 21 Feb 2022 12:39:46 GMT

Thanks for the update.

Is there any changes in sizing and memory for spark 2.4?

Re: upgrading spark to 2.4

RangaReddy — Tue, 22 Feb 2022 03:39:09 GMT

Please go through the following article.

https://community.cloudera.com/t5/Community-Articles/Spark-Memory-Management/ta-p/317794

Unified Memory Manager is introduced in Spark 1.6 onwards. There is no much changes after Unified. Spark 3 has some changes in memory management.

Re: upgrading spark to 2.4

Rajeshhadoop — Wed, 23 Feb 2022 05:36:15 GMT

Thanks for the update.

One more thing need clarification. While looking into the Spark unsupported features in CDP private cloud, i can see below statement

"Using the JDBC Datasource API to access Hive or Impala is not supported"

Can you please explain this? If JDBC datasource is not supported, can't we access hive? Is this true?

Is there any fix for this.

Appreciate your quick response.

Re: upgrading spark to 2.4

RangaReddy — Wed, 23 Feb 2022 06:35:46 GMT

Hi @Rajeshhadoop

I think it is the not right way to ask set of questions in single community article. Please create a new thread for any kind of questions.