Spark Python Supportability Matrix

[#1]

Spark Python Supportability Matrix

The Spark Python Supportability Matrix serves as an essential tool for determining which Python versions are compatible with specific Spark versions. This matrix provides a detailed overview of the compatibility levels for various Python versions across different Spark releases.

Spark Version	Python Min Supported Version	Python Max Supported Version	Python v 2.7	Python v3.4	Python v3.5	Python v 3.6	Python v3.7	Python v3.8	Python v3.9	Python v3.10	Python v3.11
3.5.2	3.8	3.11	No	No	No	No	No	Yes	Yes	Yes	Yes
3.5.1	3.8	3.11	No	No	No	No	No	Yes	Yes	Yes	Yes
3.5.0	3.8	3.11	No	No	No	No	No	Yes	Yes	Yes	Yes
3.4.2	3.7	3.11	No	No	No	No	Yes	Yes	Yes	Yes	Yes
3.4.1	3.7	3.11	No	No	No	No	Yes	Yes	Yes	Yes	Yes
3.4.0	3.7	3.11	No	No	No	No	Yes	Yes	Yes	Yes	Yes
3.3.3	3.7	3.10	No	No	No	No	Yes	Yes	Yes	Yes	No
3.3.2	3.7	3.10	No	No	No	No	Yes	Yes	Yes	Yes	No
3.3.1	3.7	3.10	No	No	No	No	Yes	Yes	Yes	Yes	No
3.3.0	3.7	3.10	No	No	No	No	Yes	Yes	Yes	Yes	No
3.2.4	3.6	3.9	No	No	No	Yes	Yes	Yes	Yes	No	No
3.2.3	3.6	3.9	No	No	No	Yes	Yes	Yes	Yes	No	No
3.2.2	3.6	3.9	No	No	No	Yes	Yes	Yes	Yes	No	No
3.2.1	3.6	3.9	No	No	No	Yes	Yes	Yes	Yes	No	No
3.2.0	3.6	3.9	No	No	No	Yes	Yes	Yes	Yes	No	No
3.1.3	3.6	3.9	No	No	No	Yes	Yes	Yes	Yes	No	No
3.1.2	3.6	3.9	No	No	No	Yes	Yes	Yes	Yes	No	No
3.1.1	3.6	3.9	No	No	No	Yes	Yes	Yes	Yes	No	No
3.0.3	2.7/3.4	3.9	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	No
3.0.2	2.7/3.4	3.9	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	No
3.0.1	2.7/3.4	3.8	Yes	Yes	Yes	Yes	Yes	Yes	No	No	No
3.0.0	2.7/3.4	3.8	Yes	Yes	Yes	Yes	Yes	Yes	No	No	No
3.0.0	2.7/3.4	3.8	Yes	Yes	Yes	Yes	Yes	Yes	No	No	No
3.0.0	2.7/3.4	3.8	Yes	Yes	Yes	Yes	Yes	Yes	No	No	No
2.4.8	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.4.7	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.4.6	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.4.5	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.4.4	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.4.3	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.4.2	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.4.1	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.4.0	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.3.4	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.3.3	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.3.2	2.7/3.4	3.7	Yes	Yes	Yes	Yes	Yes	No	No	No	No
2.3.1	2.7/3.4	3.6	Yes	Yes	Yes	Yes	No	No	No	No	No
2.3.0	2.7/3.4	3.6	Yes	Yes	Yes	Yes	No	No	No	No	No
2.2.3	2.7/3.4	3.5	Yes	Yes	Yes	No	No	No	No	No	No
2.2.2	2.7/3.4	3.5	Yes	Yes	Yes	No	No	No	No	No	No
2.2.1	2.7/3.4	3.5	Yes	Yes	Yes	No	No	No	No	No	No
2.2.0	2.7/3.4	3.5	Yes	Yes	Yes	No	No	No	No	No	No
2.1.3	2.7/3.4	3.5	Yes	Yes	Yes	No	No	No	No	No	No
2.1.2	2.7/3.4	3.5	Yes	Yes	Yes	No	No	No	No	No	No
2.1.1	2.7/3.4	3.5	Yes	Yes	Yes	No	No	No	No	No	No

Note: The above data is generated using https://pypi.org/project/pyspark/ website. If you face any problems with supported python environment share in comments so that we can put some notes.

[#2]

Hi,

Is anyone having issue while working on empty pyspark dataframe?

I installed spark 3.5.1 version and python 3.12.x, Having issue only on empty Dataframe.

spark 3.5.1 is working fine on python 3.11.x.

Any suggestion on python 3.12.x compatibility on spark 3.51?

[#3]

Pyspark 3.5.2  - python >= 3.8 and <=3.11
ref: https://pypi.org/project/pyspark/3.5.2/