- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
Created on 12-10-2023 09:39 PM - edited on 08-18-2024 10:48 PM by Gopinath
Spark Python Supportability Matrix
The Spark Python Supportability Matrix serves as an essential tool for determining which Python versions are compatible with specific Spark versions. This matrix provides a detailed overview of the compatibility levels for various Python versions across different Spark releases.
Spark Version | Python Min Supported Version | Python Max Supported Version | Python v 2.7 | Python v3.4 | Python v3.5 | Python v 3.6 | Python v3.7 | Python v3.8 | Python v3.9 | Python v3.10 | Python v3.11 |
3.5.2 | 3.8 | 3.11 | No | No | No | No | No | Yes | Yes | Yes | Yes |
3.5.1 | 3.8 | 3.11 | No | No | No | No | No | Yes | Yes | Yes | Yes |
3.5.0 | 3.8 | 3.11 | No | No | No | No | No | Yes | Yes | Yes | Yes |
3.4.2 | 3.7 | 3.11 | No | No | No | No | Yes | Yes | Yes | Yes | Yes |
3.4.1 | 3.7 | 3.11 | No | No | No | No | Yes | Yes | Yes | Yes | Yes |
3.4.0 | 3.7 | 3.11 | No | No | No | No | Yes | Yes | Yes | Yes | Yes |
3.3.3 | 3.7 | 3.10 | No | No | No | No | Yes | Yes | Yes | Yes | No |
3.3.2 | 3.7 | 3.10 | No | No | No | No | Yes | Yes | Yes | Yes | No |
3.3.1 | 3.7 | 3.10 | No | No | No | No | Yes | Yes | Yes | Yes | No |
3.3.0 | 3.7 | 3.10 | No | No | No | No | Yes | Yes | Yes | Yes | No |
3.2.4 | 3.6 | 3.9 | No | No | No | Yes | Yes | Yes | Yes | No | No |
3.2.3 | 3.6 | 3.9 | No | No | No | Yes | Yes | Yes | Yes | No | No |
3.2.2 | 3.6 | 3.9 | No | No | No | Yes | Yes | Yes | Yes | No | No |
3.2.1 | 3.6 | 3.9 | No | No | No | Yes | Yes | Yes | Yes | No | No |
3.2.0 | 3.6 | 3.9 | No | No | No | Yes | Yes | Yes | Yes | No | No |
3.1.3 | 3.6 | 3.9 | No | No | No | Yes | Yes | Yes | Yes | No | No |
3.1.2 | 3.6 | 3.9 | No | No | No | Yes | Yes | Yes | Yes | No | No |
3.1.1 | 3.6 | 3.9 | No | No | No | Yes | Yes | Yes | Yes | No | No |
3.0.3 | 2.7/3.4 | 3.9 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No |
3.0.2 | 2.7/3.4 | 3.9 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No |
3.0.1 | 2.7/3.4 | 3.8 | Yes | Yes | Yes | Yes | Yes | Yes | No | No | No |
3.0.0 | 2.7/3.4 | 3.8 | Yes | Yes | Yes | Yes | Yes | Yes | No | No | No |
3.0.0 | 2.7/3.4 | 3.8 | Yes | Yes | Yes | Yes | Yes | Yes | No | No | No |
3.0.0 | 2.7/3.4 | 3.8 | Yes | Yes | Yes | Yes | Yes | Yes | No | No | No |
2.4.8 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.4.7 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.4.6 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.4.5 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.4.4 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.4.3 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.4.2 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.4.1 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.4.0 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.3.4 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.3.3 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.3.2 | 2.7/3.4 | 3.7 | Yes | Yes | Yes | Yes | Yes | No | No | No | No |
2.3.1 | 2.7/3.4 | 3.6 | Yes | Yes | Yes | Yes | No | No | No | No | No |
2.3.0 | 2.7/3.4 | 3.6 | Yes | Yes | Yes | Yes | No | No | No | No | No |
2.2.3 | 2.7/3.4 | 3.5 | Yes | Yes | Yes | No | No | No | No | No | No |
2.2.2 | 2.7/3.4 | 3.5 | Yes | Yes | Yes | No | No | No | No | No | No |
2.2.1 | 2.7/3.4 | 3.5 | Yes | Yes | Yes | No | No | No | No | No | No |
2.2.0 | 2.7/3.4 | 3.5 | Yes | Yes | Yes | No | No | No | No | No | No |
2.1.3 | 2.7/3.4 | 3.5 | Yes | Yes | Yes | No | No | No | No | No | No |
2.1.2 | 2.7/3.4 | 3.5 | Yes | Yes | Yes | No | No | No | No | No | No |
2.1.1 | 2.7/3.4 | 3.5 | Yes | Yes | Yes | No | No | No | No | No | No |
Created on 07-03-2024 04:20 AM
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
Hi,
Is anyone having issue while working on empty pyspark dataframe?
I installed spark 3.5.1 version and python 3.12.x, Having issue only on empty Dataframe.
spark 3.5.1 is working fine on python 3.11.x.
Any suggestion on python 3.12.x compatibility on spark 3.51?
Created on 08-18-2024 10:11 PM
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
Pyspark 3.5.2 - python >= 3.8 and <=3.11
ref: https://pypi.org/project/pyspark/3.5.2/