How to enable Spark 3.5 as a Runtime Addon for Cloudera AI Workbench sessions?

[#1]

Hi all,

We've upgraded our base cluster (Cloudera Runtime 7.1.9) to CDS 3.5
via the new add-on parcel — that part is working fine.

Now we're trying to make Spark 3.5 available as a selectable Runtime
Addon inside Cloudera AI Workbench sessions, but we're stuck.

Our setup:
- Cloudera AI Workbench version: 2.0.44-h1-b4
- CDP Private Cloud Data Services: 1.5.4-h10-b2
- Deployment type: ECS
- We use Spark Pushdown (sessions submit jobs that run on the base
cluster via YARN)

What we've found so far:
- Currently only Spark versions up to 3.3.2 show up in Site
Administration > Runtime > Runtime Addons
- There's a SPARK3 3.5.4 parcel visible in Cloudera Manager under
our CAI cluster, listed as "Available Remotely" — not yet
downloaded/activated

Has anyone gotten Spark 3.5 working as a Runtime Addon on a similar
setup? Specifically:
1. Is downloading/activating that SPARK3 parcel the right approach,
or does it need to be "linked" to the workbench some other way?
2. Does this require a minimum Workbench version?
3. Any pushdown-specific gotchas with Spark 3.5?

Thanks in advance!

[#2]

Hello @DevOpsWorld, 

Thanks for reaching out on the community. It's always a pleasure. 

The problem here is the DS version you have. Data Services 1.5.4 only supports Spark 2.4.7 and 3.2.3: 

https://docs.cloudera.com/machine-learning/1.5.4/spark/topics/ml-spark-supported-versions.html

To be able to use Spark 3.5 you need to upgrade to DS 1.5.5: 

https://docs.cloudera.com/machine-learning/1.5.5/spark/topics/ml-spark-supported-versions.html

Hope this helps to clarify the situation. 

[#3]

@DevOpsWorld Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  Thanks.