How to enable Spark 3.5 as a Runtime Addon for Cloudera AI Workbench sessions?

[#1]

Hi all,

We've upgraded our base cluster (Cloudera Runtime 7.1.9) to CDS 3.5
via the new add-on parcel — that part is working fine.

Now we're trying to make Spark 3.5 available as a selectable Runtime
Addon inside Cloudera AI Workbench sessions, but we're stuck.

Our setup:
- Cloudera AI Workbench version: 2.0.44-h1-b4
- CDP Private Cloud Data Services: 1.5.4-h10-b2
- Deployment type: ECS
- We use Spark Pushdown (sessions submit jobs that run on the base
cluster via YARN)

What we've found so far:
- Currently only Spark versions up to 3.3.2 show up in Site
Administration > Runtime > Runtime Addons
- There's a SPARK3 3.5.4 parcel visible in Cloudera Manager under
our CAI cluster, listed as "Available Remotely" — not yet
downloaded/activated

Has anyone gotten Spark 3.5 working as a Runtime Addon on a similar
setup? Specifically:
1. Is downloading/activating that SPARK3 parcel the right approach,
or does it need to be "linked" to the workbench some other way?
2. Does this require a minimum Workbench version?
3. Any pushdown-specific gotchas with Spark 3.5?

Thanks in advance!

[#2]

Hello @DevOpsWorld, 

Thanks for reaching out on the community. It's always a pleasure. 

The problem here is the DS version you have. Data Services 1.5.4 only supports Spark 2.4.7 and 3.2.3: 

https://docs.cloudera.com/machine-learning/1.5.4/spark/topics/ml-spark-supported-versions.html

To be able to use Spark 3.5 you need to upgrade to DS 1.5.5: 

https://docs.cloudera.com/machine-learning/1.5.5/spark/topics/ml-spark-supported-versions.html

Hope this helps to clarify the situation.