Member since
01-31-2026
3
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 9 | 02-02-2026 07:14 PM |
02-02-2026
07:14 PM
Seems like the message was there all along when we choose the runtime, e.g. Jupyterlab -> Python xx -> Edition: Nvidia GPU, and enable Spark; the message will appear: "Spark is not compatible with the selected Edition. If you enable Spark for the session, it can be used independently but it will not be accelerated" I didnt see the warning message before because have only allowed our own customized runtime, which didn't display this warning message.
... View more
02-02-2026
04:56 PM
Thanks for the reply @vafs ; I can't be sure that was the definite answer though, because it is combining answers from two different sources: - first one is about supporting spark (nothing about gpu mentioned). - second one is about gpu (nothing about spark mentioned). And when I tried a Spark code that works in YARN+GPU, with slight modification to fit into CML, it just didnt go well. Not sure if I've done something wrong, that's why I am looking for a definite answer, probably with some github example like what Cloudera have provided for pytorch and tensorflow for CML. Hence, me raising this question. Somehow, I kind of remembering seeing somewhere that in CDE, it is only Technical Preview, but in CML it is not yet; but can't seem to be able to find where was that page :).
... View more
01-31-2026
09:59 PM
Hi everyone. Just want to check as of latest CML version, has there been any support for using GPU for Spark in CML ? I have seen: 1. GPU used for Spark in CDE, or 2. GPU used in general Python project ( using torch, tensorflow, rapids' cudf/cuml) in CML, but can't find any for Spark in CML. If there is no such support; is there any specific reason why it cannot be done while it can be done for general Spark on k8s. Thanks for any info...
... View more
Labels: