Migrating workloads from Spark 2 to Spark 3

[#1]

Dear Cloudera team,

@Bharati @jagadeesan 

We are currently using Cloudera 7.11.3, and Spark actions added via the UI are still being treated as Spark2. We would like to use Spark3 by default instead. How can we migrate or configure the system to ensure Spark3 is used for these actions?

Best Regards,

Modhi

[#2]

Do you have any insights here @smdas @Gopinath ? Thanks!

[#3 · Accepted solution]

@Malrashed, If you are using Cloudera Runtime 7.1.9 then you can use either CDS 3.3 or CDS 3.5 Powered by Apache Spark as an add-on service. For more details, you can refer to this document. Please note, CDS 3.3 Powered by Apache Spark 3.3.x and CDS 3.5 Powered by Apache Spark 3.5.x are distributed as a parcel (Refer here for additional download details). There are no external Custom Service Descriptors (CSD) for Livy for Spark3 or Spark3 because they are already part of Cloudera Manager 7.11.3.

In Cloudera Runtime 7.1.9, Spark 2 is the default. If you need to use Spark 3, it must be added as an add-on service. Note that Spark 2 is deprecated in Cloudera Runtime 7.1.9. Starting with Cloudera Runtime 7.3.x, Spark 3 becomes the default version