Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

upgrading spark to 2.4

avatar
Explorer

Dear All,

 

On one particular cluster we have spark 2.3 running(CDH 5.15.1). can it be possible to only upgrade spark to 2.4 ? by remaining on the same CDH version(5.15.1). will it compatible with existing service ?. kindly suggest  

1 ACCEPTED SOLUTION

avatar
Super Collaborator

Hi @javidshaik 

 

Yes based on cloudera documentation, it is not supported multiple versions under same Cloudera Manager Server.

View solution in original post

12 REPLIES 12

avatar
Super Collaborator

Please go through the following article. 

 

https://community.cloudera.com/t5/Community-Articles/Spark-Memory-Management/ta-p/317794

 

Unified Memory Manager is introduced in Spark 1.6 onwards. There is no much changes after Unified. Spark 3 has some changes in memory management.

 

avatar

Thanks for the update.

One more thing need clarification. While looking into the Spark unsupported features in CDP private cloud, i can see below statement 

"Using the JDBC Datasource API to access Hive or Impala is not supported"

Can you please explain this? If JDBC datasource is not supported, can't we access hive? Is this true?

Is there any fix for this.

Appreciate your quick response.

 

avatar
Super Collaborator

Hi @Rajeshhadoop 

 

I think it is the not right way to ask set of questions in single community article. Please create a new thread for any kind of questions.