Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
avatar
Super Collaborator

Spark and Java versions Supportability Matrix

1. Introduction:

Apache Spark is a powerful open-source distributed computing system widely used for big data processing and analytics. However, choosing the right Java version for your Spark application is crucial for optimal performance, security, and compatibility

This article dives deep into the officially supported Java versions for Spark, along with helpful advice on choosing the right one for your project.

2. Matrix Table

Spark Version

Supported Java Version(s)

Java 8

Java 11

Java 17

Java 21

Deprecated Java Version(s)

3.5.1

Java 8*/11/17

Yes

Yes

Yes

No

Java 8 prior to version 8u371 support is deprecated

3.5.0

Java 8*/11/17

Yes

Yes

Yes

No

Java 8 prior to version 8u371 support is deprecated

3.4.2

Java 8*/11/17

Yes

Yes

Yes

No

Java 8 prior to version 8u362 support is deprecated

3.4.1

Java 8*/11/17

Yes

Yes

Yes

No

Java 8 prior to version 8u362 support is deprecated

3.4.0

Java 8*/11/17

Yes

Yes

Yes

No

Java 8 prior to version 8u362 support is deprecated

3.3.3

Java 8*/11/17

Yes

Yes

Yes

No

Java 8 prior to version 8u201 support is deprecated

3.3.2

Java 8*/11/17

Yes

Yes

Yes

No

Java 8 prior to version 8u201 support is deprecated

3.3.1

Java 8*/11/17

Yes

Yes

Yes

No

Java 8 prior to version 8u201 support is deprecated

3.3.0

Java 8*/11/17^

Yes

Yes

No

No

Java 8 prior to version 8u201 support is deprecated

3.2.4

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u201 support is deprecated

3.2.3

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u201 support is deprecated

3.2.2

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u201 support is deprecated

3.2.1

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u201 support is deprecated

3.2.0

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u201 support is deprecated

3.1.3

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u92 support is deprecated

3.1.2

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u92 support is deprecated

3.1.1

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u92 support is deprecated

3.0.3

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u92 support is deprecated

3.0.2

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u92 support is deprecated

3.0.1

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u92 support is deprecated

3.0.0

Java 8*/11

Yes

Yes

No

No

Java 8 prior to version 8u92 support is deprecated

2.4.8

Java 8*

Yes

No

No

No

 

2.4.7

Java 8*

Yes

No

No

No

 

2.4.6

Java 8*

Yes

No

No

No

 

2.4.5

Java 8*

Yes

No

No

No

 

2.4.4

Java 8*

Yes

No

No

No

 

2.4.3

Java 8*

Yes

No

No

No

 

2.4.2

Java 8*

Yes

No

No

No

 

2.4.1

Java 8*

Yes

No

No

No

 

2.4.0

Java 8*

Yes

No

No

No

 

 

 

 

* means Cloudera recommended Java version.
^ means Upstream Spark is supported.

 

 

Note: According to the Cloudera documentation, Spark 3.3.0 only supports Java 8 and 11. However, the official Spark documentation lists Java 8, 11, and 17 as compatible versions.

References:

  1. Apache Spark - A Unified engine for large-scale data analytics
  2. CDS 3.3 Powered by Apache Spark Requirements
  3. SPARK-24417
  4. Support Matrix Cloudera

3. Problems Arising from Unsupported Spark & Java Versions

Utilizing incompatible or unsupported versions of Spark and Java can introduce various challenges and impediments in the operation of your Spark environment.

  1. Performance Degradation: The usage of an incompatible Java version could lead to performance degradation or inefficiencies. This is attributable to the inability to leverage the latest optimizations and features provided by newer Java releases, resulting in the suboptimal performance of Spark jobs.
  2. Compatibility Issues: Spark's functionality may be compromised or rendered unstable when interfacing with specific versions of Java. This can manifest as unexpected errors or failures during runtime, hindering the smooth execution of Spark applications.

  3. Feature Limitations: Newer iterations of Spark may rely on features or enhancements exclusive to certain Java versions. Employing outdated or unsupported Java versions may curtail your ability to exploit these advanced features, constraining the capabilities and functionalities of your Spark applications.

4. End of Life (EOL) dates for Java versions:

 

Java Version 

EOL Date

1

Java 8

31, December 2020 (Public Updates), Still supported with Long Term Support (LTS) until December 2030.

2

Java 11

30, September 2023 (Public Updates), Still supported with Long Term Support (LTS) until January 2032.

3

Java 17 

30, September 2026 (Public Updates), Long Term Support (LTS) until September 2029.

4

Java 21

30, September 2028 (Public Updates), Long Term Support (LTS) until September 2031.

Reference(s):

  1. Oracle Java SE Support Roadmap
  2. Java version history

5. JDK & Scala compatibility

Minimum Scala versions:

JDK VersionScala 3Scala 2.13Scala 2.12Scala 2.11
22 (ea)3.3.22.13.122.12.19 
21 (LTS)3.3.12.13.112.12.18 
203.3.02.13.112.12.18 
193.2.02.13.92.12.16 
183.1.32.13.72.12.15 
17 (LTS)3.0.02.13.62.12.15 
11 (LTS)3.0.02.13.02.12.42.11.12
8 (LTS)3.0.02.13.02.12.02.11.0

`*` = forthcoming; support available in nightly builds

Thank you for taking the time to read this article. We hope you found it informative and helpful in enhancing your understanding of the topic. If you have any questions or feedback, please feel free to reach out to me. Remember, your support motivates us to continue creating valuable content. If this article helped you, please consider giving it a like and providing a kudos. We appreciate your support!

3,404 Views