Spark Python Integration Test Result Exceptions
Spark Python Integration Test Result Exceptions
In this article, just I talk about exceptions and their Python and Spark versions. Keep on watching this article where I will add some more exceptions and solutions.
1. TypeError: an integer is required (got type bytes)
Exception:
Traceback (most recent call last):
File "/opt/pyspark_udf_example.py", line 3, in <module>
from pyspark.sql import SparkSession
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 46, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 31, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "/opt/spark/python/lib/pyspark.zip/pyspark/accumulators.py", line 97, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 146, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 127, in _make_cell_set_template_code
TypeError: an integer is required (got type bytes)
Python Versions:
- Python 3.8
- Python 3.9
Spark Versions:
- Spark Version: 2.3.0
- Spark Version: 2.3.1
- Spark Version: 2.3.2
- Spark Version: 2.3.3
- Spark Version: 2.3.4
- Spark Version: 2.4.0
- Spark Version: 2.4.1
- Spark Version: 2.4.2
- Spark Version: 2.4.3
- Spark Version: 2.4.4
- Spark Version: 2.4.5
- Spark Version: 2.4.6
- Spark Version: 2.4.7
- Spark Version: 2.4.8
2. TypeError: 'bytes' object cannot be interpreted as an integer
Exception:
Traceback (most recent call last):
File "/opt/pyspark_udf_example.py", line 3, in <module>
from pyspark.sql import SparkSession
File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 46, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 31, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/accumulators.py", line 97, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 146, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 127, in _make_cell_set_template_code
TypeError: 'bytes' object cannot be interpreted as an integer
Python Versions:
- Python 3.10
Spark Versions:
- Spark Version: 2.3.0
- Spark Version: 2.3.1
- Spark Version: 2.3.2
- Spark Version: 2.3.3
- Spark Version: 2.3.4
- Spark Version: 2.4.0
- Spark Version: 2.4.1
- Spark Version: 2.4.2
- Spark Version: 2.4.3
- Spark Version: 2.4.4
- Spark Version: 2.4.5
- Spark Version: 2.4.6
- Spark Version: 2.4.7
- Spark Version: 2.4.8
3. TypeError: code expected at least 16 arguments, got 15
Exception:
Traceback (most recent call last):
File "/opt/pyspark_udf_example.py", line 3, in <module>
from pyspark.sql import SparkSession
File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 46, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 31, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/accumulators.py", line 97, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 146, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 127, in _make_cell_set_template_code
TypeError: code expected at least 16 arguments, got 15
Python Versions:
- Python 3.11
Spark Versions:
- Spark Version: 2.3.0
- Spark Version: 2.3.1
- Spark Version: 2.3.2
- Spark Version: 2.3.3
- Spark Version: 2.3.4
- Spark Version: 2.4.0
- Spark Version: 2.4.1
- Spark Version: 2.4.2
- Spark Version: 2.4.3
- Spark Version: 2.4.4
- Spark Version: 2.4.5
- Spark Version: 2.4.6
- Spark Version: 2.4.7
- Spark Version: 2.4.8
4. TypeError: code() argument 13 must be str, not int
Exception:
Traceback (most recent call last):
File "/opt/pyspark_udf_example.py", line 3, in <module>
from pyspark.sql import SparkSession
File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 51, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 30, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/accumulators.py", line 97, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 71, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 209, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 172, in _make_cell_set_template_code
TypeError: code() argument 13 must be str, not int
Python Versions:
- Python 3.11
Spark Versions:
- Spark Version: 3.0.0
- Spark Version: 3.0.1
- Spark Version: 3.0.3
- Spark Version: 3.0.3
5. SyntaxError: invalid syntax
Exception:
Traceback (most recent call last):
File "/opt/pyspark_udf_example.py", line 3, in <module>
from pyspark.sql import SparkSession
File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 53, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 34, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/java_gateway.py", line 31, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/find_spark_home.py", line 68
print("Could not find valid SPARK_HOME while searching {0}".format(paths), file=sys.stderr)
^
SyntaxError: invalid syntax
Python Versions:
- Python 2.7
Spark Versions:
- Spark Version: 3.1.1
- Spark Version: 3.1.2
- Spark Version: 3.1.3
- Spark Version: 3.2.0
- Spark Version: 3.2.1
- Spark Version: 3.2.2
- Spark Version: 3.2.3
6. ImportError: No module named 'typing'
Exception:
Traceback (most recent call last):
File "/opt/pyspark_udf_example.py", line 3, in <module>
from pyspark.sql import SparkSession
File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 53, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 34, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/java_gateway.py", line 32, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 67, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle/__init__.py", line 4, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle/cloudpickle.py", line 54, in <module>
ImportError: No module named 'typing'
Python Versions:
- Python 3.4
Spark Versions:
- Spark Version: 3.1.1
- Spark Version: 3.1.2
- Spark Version: 3.1.3
- Spark Version: 3.2.0
- Spark Version: 3.2.1
- Spark Version: 3.2.2
- Spark Version: 3.2.3
- Spark Version: 3.3.0
- Spark Version: 3.3.1
7. AttributeError: 'NoneType' object has no attribute 'items'
Exception:
Traceback (most recent call last):
File "/opt/pyspark_udf_example.py", line 3, in <module>
from pyspark.sql import SparkSession
File "<frozen importlib._bootstrap>", line 968, in _find_and_load
File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 53, in <module>
File "<frozen importlib._bootstrap>", line 968, in _find_and_load
File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
File "/opt/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 48, in <module>
File "<frozen importlib._bootstrap>", line 968, in _find_and_load
File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
File "/opt/spark/python/lib/pyspark.zip/pyspark/traceback_utils.py", line 23, in <module>
File "/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 390, in namedtuple
AttributeError: 'NoneType' object has no attribute 'items'
Python Versions:
- Python 3.5
Spark Versions:
- Spark Version: 3.1.1
- Spark Version: 3.1.2
- Spark Version: 3.1.3
- Spark Version: 3.2.0
- Spark Version: 3.2.1
- Spark Version: 3.2.2
- Spark Version: 3.2.3
- Spark Version: 3.3.2
- Spark Version: 3.4.0
8. SyntaxError: invalid syntax
Exception:
Traceback (most recent call last):
File "/opt/pyspark_udf_example.py", line 3, in <module>
from pyspark.sql import SparkSession
File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 71
def since(version: Union[str, float]) -> Callable[[F], F]:
^
SyntaxError: invalid syntax
Python Versions:
- Python 2.7
Spark Versions:
- Spark Version: 3.3.0
- Spark Version: 3.3.1
- Spark Version: 3.3.2
- Spark Version: 3.4.0
9. SyntaxError: invalid syntax
Exception:
Traceback (most recent call last):
File "/opt/pyspark_udf_example.py", line 3, in <module>
from pyspark.sql import SparkSession
File "<frozen importlib._bootstrap>", line 968, in _find_and_load
File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 53, in <module>
File "<frozen importlib._bootstrap>", line 968, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 896, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1171, in find_spec
File "<frozen importlib._bootstrap_external>", line 1147, in _get_spec
File "<frozen importlib._bootstrap_external>", line 1128, in _legacy_get_spec
File "<frozen importlib._bootstrap>", line 444, in spec_from_loader
File "<frozen importlib._bootstrap_external>", line 565, in spec_from_file_location
File "/opt/spark/python/lib/pyspark.zip/pyspark/conf.py", line 110
_jconf: Optional[JavaObject]
^
SyntaxError: invalid syntax
Python Versions:
- Python 3.5
Spark Versions:
- Spark Version: 3.3.0
- Spark Version: 3.3.1
- Spark Version: 3.3.2
- Spark Version: 3.4.0
Note: The above all exceptions occurred while testing the pyspark code (sample udf) example with different Python versions.
