Spark jobs in Cloudera AI (CML) fail with 'ClassCastException: List$SerializationProxy' a JDK mismatch between driver and executor pods

[#1]

Can you rephrase this title in this format: Error... occurs when... in... Spark jobs in Cloudera AI (CML) fail with 'ClassCastException: List$SerializationProxy' a JDK mismatch between driver and executor pods

A JDK version mismatch between driver and executor pods

Environment

Component	Version
CDP Private Cloud Base	7.3.2 (upgraded in place from 7.3.1)
Cloudera Manager	7.13.2
Cloudera Data Services	1.5.5-h2100-b23
Cloudera AI Workbench (CML)	2.0.51-h2100-b26, on OpenShift Container Platform
Runtime	PBJ Workbench · Python 3.11 · Standard · 2026.08
Runtime image	ml-runtime-pbj-workbench-python3.11-standard:2026.08.1-b5
Spark add-on	Spark 3.5.4 – CDP 7.3.1.400  (latest available; no 7.3.2 add-on offered)
Table format	Apache Iceberg V2, Hive external warehouse

Problem

In a CML session with Spark enabled, every driver-side operation succeeds:

spark.sql("SHOW DATABASES").show()
spark.sql("SHOW TABLES").show()
df = spark.sql("SELECT * FROM my_db.my_iceberg_table LIMIT 20")   # no error

The moment an action distributes work to executors, the job dies:

df.show()
org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 0.0 failed 4 times, most recent failure:
Lost task 0.3 in stage 0.0 (TID 3) (10.128.14.51 executor 1):
java.lang.ClassCastException: cannot assign instance of
scala.collection.immutable.List$SerializationProxy to field
org.apache.spark.sql.execution.datasources.v2.DataSourceRDDPartition.inputPartitions
of type scala.collection.Seq in instance of
org.apache.spark.sql.execution.datasources.v2.DataSourceRDDPartition

The message points at Iceberg, Scala, or a jar conflict. It is none of those.

Analysis

1. The failure is in task deserialization, not the query: SHOW DATABASES, SHOW TABLES, and building the DataFrame all succeeded — those are driver-side. The first thing to cross the driver → executor boundary was df.show(), and that is what failed. The problem is serialization between two JVMs, not catalog access, Kerberos, Ranger, or KMS.
2. Read the JVM version straight out of the stack trace: This is the step that cracks it. Compare frames from the two sides of the same trace: 

   Driver frames:
   
   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl
   
          .invoke(NativeMethodAccessorImpl.java:77)
   
   at java.base/java.lang.Thread.run(Thread.java:840)
   
   Executor frames:
   
   at java.io.ObjectStreamClass$FieldReflector
   
          .setObjFieldValues(ObjectStreamClass.java:2301)
   
   at java.lang.Thread.run(Thread.java:750)
   ​

   The java.base/ module prefix appears only on JDK 9 and later. Thread.java:840 is JDK 17; Thread.java:750 is JDK 8.
   
   The driver was running Java 17 and the executors were running Java 8: scala.collection.immutable.List$SerializationProxy resolves differently across JDK versions during readResolve, and DataSourceRDDPartition.inputPartitions is one of the few places Spark serializes a raw Scala List into a Seq-typed field. That is why the symptom appears on DataSource V2 scans (Iceberg and other V2 sources) while some plain RDD operations still succeed — and why it looks Iceberg-specific when it isn't.
   
   
3. Confirm it from inside a task: Stack-trace reading is suggestive; this is proof. Run the probe as a Spark task so it executes in the executor pod:

   def probe(_):
   
       import os, glob, sys
   
       jars = glob.glob("/opt/spark/jars/spark-core*")
   
       return {
   
           "JAVA_HOME": os.environ.get("JAVA_HOME", "unset"),
   
           "jvms": glob.glob("/usr/lib/jvm/*"),
   
           "spark_core": [os.path.basename(p) for p in jars],
   
           "python": sys.version.split()[0],
   
       }
   
   
   
   print(spark.sparkContext.parallelize([1], 1).map(probe).collect())
   
   Result:
   
   [{'JAVA_HOME': '/usr/lib/jvm/java-8-openjdk-amd64',
   
     'jvms': ['/usr/lib/jvm/java-8-openjdk-amd64',
   
              '/usr/lib/jvm/java-17-openjdk-amd64'],
   
     'spark_core': ['spark-core_2.12-3.5.4.1.24.731.1000-22.jar'],
   
     'python': '3.11.15'}]
   
   Against the driver:
   
   !echo $JAVA_HOME      # /usr/lib/jvm/java-17-openjdk-amd64
   
   !java -version        # openjdk version "17.0.15"
   
   

   Same Spark jar. Same Python. Same container image — spark.kubernetes.executor.container.image was unset, so executors inherit the driver's image. Both JDKs are present in that image; the executor simply points JAVA_HOME at the older one.
   
   Pro tip: do not use subprocess.run(["java", "-version"]). java is not on PATH in the executor container, and the resulting FileNotFoundError masks the real answer. Read JAVA_HOME and glob /usr/lib/jvm/ instead.
   
   
4. Root cause: runtime and Spark add-on from different lines: The PBJ Workbench 2026.08 runtime image defaults to JDK 17. The Spark 3.5.4 – CDP 7.3.1.400 add-on dates from the JDK 8 runtime line and its executor environment still sets JAVA_HOME to the Java 8 path. Pairing them puts driver and executors on different JVMs inside a single image.
   
   Note the platform context. This cluster was upgraded in place from CDP 7.3.1 to 7.3.2, but 7.3.1.400 remains the newest Spark add-on offered — there is no 7.3.2-matched add-on to select. The likely reason is version skew across the stack: Cloudera Data Services is still at 1.5.5, a 7.3.1-era release, while Base moved to 7.3.2. The Spark add-on catalog available to a workspace tracks the Data Services release rather than the Base cluster version, so upgrading Base alone does not surface newer add-ons.
   
   The practical consequence is that users cannot resolve this by choosing a newer add-on. The only lever available is the Runtime version.

Fix

Select a Runtime version from the same line as the Spark add-on. In Start A New Session, change the Runtime Version dropdown from 2026.08 to 2025.09, leaving the Spark add-on as Spark 3.5.4 – CDP 7.3.1.400. Driver and executor JVMs then match and DSv2 scans complete normally.

No Spark configuration change was required.

Verified runtime pairings (Spark 3.5.4 – CDP 7.3.1.400)

Runtime version	Driver JVM	Executor JVM	Result
2025.09	matched	matched	Works
2026.01	—	—	not yet tested
2026.04	—	—	not yet tested
2026.08	Java 17	Java 8	Fails — ClassCastException

If you are on a version not listed, run the check below rather than assuming.

Verification cell — run this first, every time

print("driver:", spark.sparkContext._jvm.System.getProperty("java.version"))

print("executor:", spark.sparkContext.parallelize([1], 1)

      .map(lambda _: __import__("os").environ.get("JAVA_HOME")).collect())

What matters is that the two match. Both Java 8 is fine. Both Java 17 is fine. A split is not.

If you cannot change the Runtime version

Overriding the executor environment is a possible stopgap:

.config("spark.executorEnv.JAVA_HOME", "/usr/lib/jvm/java-17-openjdk-amd64")

Confirm with the verification cell — an environment variable supplied by the executor pod template can take precedence over what Spark injects, in which case the override silently does nothing. This is a workaround, not a repair.

A note on Iceberg jars

During debugging, spark.driver.extraClassPath and spark.executor.extraClassPath were also pointed at the Iceberg runtime jar under /runtime-addons/<addon>/opt/spark/optional-lib/. That change was made in the same step as the Runtime version change and was not independently isolated. The executor probe showed the addon directory and the Iceberg jar already mounted in the executor pod, so on this workspace the classpath configuration may have been unnecessary.

Change the Runtime version first. Add classpath configuration only if scans still fail.

For reference, the addon directory name is generated per workspace, so glob it rather than hardcoding:

import glob

ICEBERG = glob.glob(

    "/runtime-addons/*/opt/spark/optional-lib/iceberg-spark-runtime-3.5_2.12-*.jar")[0]

Takeaways

• A ClassCastException on List$SerializationProxy during task deserialization is a JVM version mismatch — not a Scala, Iceberg, or jar-conflict problem.
• Each side's JDK version is readable directly from the stack trace: the java.base/ module prefix means JDK 9+, and Thread.java line numbers differ per release.
• In CML, driver and executor can run the same container image and still differ in JAVA_HOME. Probe the executor from inside a task rather than assuming symmetry.
• getOrCreate() returns the existing SparkContext and silently discards new configuration. Restart the session when changing spark.executorEnv.*, spark.jars, or extraClassPath — otherwise you will be testing a config that never applied.
• Newest is not safest. When a platform upgrade outpaces the available Spark add-ons, the Runtime version becomes the compatibility lever — check the pairing before debugging anything else.

DISCLAIMER:  An external user contributed to this article. Cloudera may not verify that the steps may be applicable for all use cases and may be very specific to a particular distribution. Please follow with caution and at your own risk. If needed, raise a support case to get confirmation.