Support Questions

Find answers, ask questions, and share your expertise

upgrading spark to 2.4

New Contributor

Dear All,

 

On one particular cluster we have spark 2.3 running(CDH 5.15.1). can it be possible to only upgrade spark to 2.4 ? by remaining on the same CDH version(5.15.1). will it compatible with existing service ?. kindly suggest  

1 ACCEPTED SOLUTION

Expert Contributor

Hi @javidshaik 

 

Yes based on cloudera documentation, it is not supported multiple versions under same Cloudera Manager Server.

View solution in original post

12 REPLIES 12

Expert Contributor

Hi @javidshaik 

 

CDH 5.x and HDP 2.X version clusters has reached end of life support. Better upgrade your cluster to CDH 6.X/CDP 7.X version.

 

Both CDH 6.X and CDP 7.X clusters will support Spark 2.4 versions.

 

Please refer following documentation:

https://www.cloudera.com/legal/policies/support-lifecycle-policy.html

 

New Contributor

Hi @RangaReddy

 

Yeah true i agree. thanks, but just wanted to know is it feasible or not to only update to spark2.4 on the cluster  ? or will there be any compatibility issues.  Appreciate your response 

Expert Contributor

Hi @javidshaik 

 

I have checked with internal team. We can migrate Spark version from Spark 2.3 to 2.4 mentioned the details in below document.

 

2.4 Release 2CDH 5.10 and any higher CDH 5.x versions
2.4 Release 1CDH 5.10 and any higher CDH 5.x versions

 

https://docs.cloudera.com/documentation/spark2/latest/topics/spark2_requirements.html

 

But Spark 2.3 -> 2.4 version changes have higher potential of risks.

 

If you are satisfied with my answer please Accept as solution.

New Contributor

hi @RangaReddy 

 

thanks for the below details.

 

Basically i have two(PROD & Stage) CDH (5.15.1) clusters with spark 2.3(2.3.0.cloudera4-1.cdh5.13.3.p0.611179) running on each cluster  under one CM. so i wanted to upgrade to spark2.4 only on one(stage) CDH(5.15.1) cluster. so as per below documentation link provided its being said

 

" All CDH clusters managed by the same Cloudera Manager Server must use exactly the same version of CDS Powered by Apache Spark. For example, you cannot use the built-in CDH Spark service, a CDS 2.1 service, and a CDS 2.2 service. You must choose only one CDS 2 Powered by Apache Spark release"

 

So i believe we cannot perform upgrade on this right. kindly confirm 

Expert Contributor

Hi @javidshaik 

 

Yes based on cloudera documentation, it is not supported multiple versions under same Cloudera Manager Server.

New Contributor

ok thanks for the confirmation @RangaReddy 

Hi 

 

As mentioned below, Spark 2.3 -> 2.4 version changes have higher potential of risks.

May i know what are the those risks. It would be more helpful if you share right link for these and also share what the considerations need to be taken while upgrading form spark 1.6 to 2.4 and spark 2.3 to spark 2.4

 

Thanks

Rajesh

Expert Contributor

Thanks for the update.

Is there any changes in sizing and memory for spark 2.4?

Expert Contributor

Please go through the following article. 

 

https://community.cloudera.com/t5/Community-Articles/Spark-Memory-Management/ta-p/317794

 

Unified Memory Manager is introduced in Spark 1.6 onwards. There is no much changes after Unified. Spark 3 has some changes in memory management.

 

Thanks for the update.

One more thing need clarification. While looking into the Spark unsupported features in CDP private cloud, i can see below statement 

"Using the JDBC Datasource API to access Hive or Impala is not supported"

Can you please explain this? If JDBC datasource is not supported, can't we access hive? Is this true?

Is there any fix for this.

Appreciate your quick response.

 

Expert Contributor

Hi @Rajeshhadoop 

 

I think it is the not right way to ask set of questions in single community article. Please create a new thread for any kind of questions.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.