Hi all, We've upgraded our base cluster (Cloudera Runtime 7.1.9) to CDS 3.5 via the new add-on parcel — that part is working fine. Now we're trying to make Spark 3.5 available as a selectable Runtime Addon inside Cloudera AI Workbench sessions, but we're stuck. Our setup: - Cloudera AI Workbench version: 2.0.44-h1-b4 - CDP Private Cloud Data Services: 1.5.4-h10-b2 - Deployment type: ECS - We use Spark Pushdown (sessions submit jobs that run on the base cluster via YARN) What we've found so far: - Currently only Spark versions up to 3.3.2 show up in Site Administration > Runtime > Runtime Addons - There's a SPARK3 3.5.4 parcel visible in Cloudera Manager under our CAI cluster, listed as "Available Remotely" — not yet downloaded/activated Has anyone gotten Spark 3.5 working as a Runtime Addon on a similar setup? Specifically: 1. Is downloading/activating that SPARK3 parcel the right approach, or does it need to be "linked" to the workbench some other way? 2. Does this require a minimum Workbench version? 3. Any pushdown-specific gotchas with Spark 3.5? Thanks in advance! ... View more

I'm running into an issue with custom runtime addon uploads on CML 2.0.44-h1-b4 (Kubernetes). Wondering if anyone else has seen this or can confirm my fix is safe. **Problem:** When uploading a runtime addon via the API endpoint `/api/v2/runtimeaddons/custom`, it shows "Initialized" in the UI, then fails after ~5 minutes with no error message. API logs show: ``` ERROR api.runtimeaddons Finish writing Custom Runtime Addon: failed. ERROR api.runtimeaddons Finish adding new custom runtime addon failed: Failed saving addon content to NFS. ``` **What I Found:** After digging through the deployment configs, the API pod is missing the `projects-pvc` volume mount. Other pods like `ds-vfs` and `s2i-client` have it mounted at `/projects`, but the API deployment doesn't. ```bash # PVC shows it's used by ds-vfs and s2i-client, but NOT api kubectl describe pvc projects-pvc # API deployment has no projects-pvc volume kubectl get deploy/api -o yaml | grep projects # (returns nothing) ``` **Proposed Fix:** Add the volume mount to the API deployment: ```bash kubectl -n [namespace] patch deployment api --type='json' -p='[ { "op": "add", "path": "/spec/template/spec/volumes/-", "value": {"name": "projects-claim", "persistentVolumeClaim": {"claimName": "projects-pvc"}} }, { "op": "add", "path": "/spec/template/spec/containers/0/volumeMounts/-", "value": {"name": "projects-claim", "mountPath": "/projects"} } ]' ``` **Questions:** 1. Has anyone else hit this issue? 2. Is this mount supposed to be there by default? 3. Any risks to patching this in production? 4. Is this a known bug or did our installation miss something? The addon metadata DOES save to the database, but the tarball content fails to write. My tarball and metadata.json are valid per the docs. Any insights appreciated before I patch our production cluster! Thanks! ... View more