Issues running Spark in CML

ipson — Tue, 21 Apr 2026 06:24:46 GMT

I have a CML project using a JupyterLab Runtime with Python 3.10 and I want to start a Spark cluster with my CDP Datalake. I'm using the predefined Spark Data Lake Connection in CML which looks like this:

```

import cml.data_v1 as cmldata

# Sample in-code customization of spark configurations
#from pyspark import SparkContext
#SparkContext.setSystemProperty('spark.executor.cores', '1')
#SparkContext.setSystemProperty('spark.executor.memory', '2g')

CONNECTION_NAME = "hiaa-dl"
conn = cmldata.get_connection(CONNECTION_NAME)
spark = conn.get_spark_session()

# Sample usage to run query through spark
EXAMPLE_SQL_QUERY = "show databases"
spark.sql(EXAMPLE_SQL_QUERY).show()

```

When I execute this I get the error:

IllegalArgumentException: The value of property spark.app.name must not be null

I'm using the predefined spark-defaults.conf which looks like this:

```

spark.executor.memory=1g
spark.executor.cores=1
spark.yarn.access.hadoopFileSystems=abfs://[container]@[storage-account].dfs.core.windows.net

```

Is there something else I need to configure in the CML session or at the data lake level?

Re: Issues running Spark in CML

ipson — Wed, 16 Oct 2024 19:10:52 GMT

Resolved. I had ML Runtimes Addons disabled. Went into CML > Site Administrations > Settings and Under Feature Flags, unchecked the checkbox next to Allow users to Run ML Runtimes Addons.

Then, started a new session with Spark enabled

Re: Issues running Spark in CML

ipson — Thu, 17 Oct 2024 11:56:36 GMT

Correction: 'Check the checkbox to Allow users to Run ML Runtimes'

question Issues running Spark in CML in Support Questions

Issues running Spark in CML

Re: Issues running Spark in CML

Re: Issues running Spark in CML