question Re: Migrating workloads from Spark 2 to Spark 3 in Support Questions

Migrating workloads from Spark 2 to Spark 3

Malrashed — Wed, 06 Aug 2025 15:32:33 GMT

Dear Cloudera team,

We are currently using Cloudera 7.11.3, and Spark actions added via the UI are still being treated as Spark2. We would like to use Spark3 by default instead. How can we migrate or configure the system to ensure Spark3 is used for these actions?

Best Regards,

Modhi

Re: Migrating workloads from Spark 2 to Spark 3

DianaTorres — Fri, 08 Aug 2025 23:05:03 GMT

Do you have any insights here @smdas @Gopinath ? Thanks!

Re: Migrating workloads from Spark 2 to Spark 3

jagadeesan — Fri, 08 Aug 2025 23:22:26 GMT

@Malrashed, If you are using Cloudera Runtime 7.1.9 then you can use either CDS 3.3 or CDS 3.5 Powered by Apache Spark as an add-on service. For more details, you can refer to this document. Please note, CDS 3.3 Powered by Apache Spark 3.3.x and CDS 3.5 Powered by Apache Spark 3.5.x are distributed as a parcel (Refer here for additional download details). There are no external Custom Service Descriptors (CSD) for Livy for Spark3 or Spark3 because they are already part of Cloudera Manager 7.11.3.

In Cloudera Runtime 7.1.9, Spark 2 is the default. If you need to use Spark 3, it must be added as an add-on service. Note that Spark 2 is deprecated in Cloudera Runtime 7.1.9. Starting with Cloudera Runtime 7.3.x, Spark 3 becomes the default version