Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

pyspark fails with python 3.6

pyspark fails with python 3.6

Explorer

Hi All,

 

We are currently using Spark 1.6 on CDH 5.10 platform. We are currently upgrading from python 2.7 to python 3.6 using anaconda distribution. While i try to do spark-submit in client mode the process is failing giving below error -

 

File "/apps/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/lib/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 381, in namedtuple
TypeError: namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module'

 

We are not very clear about the cause of the failure. We have checked Spark documentation and it says that Spark 1.6.0 is compatible with python 3.0+. 

 

Any thoughts or suggestions on this would be helpful ?

 

Thanks

Hemil

2 REPLIES 2

Re: pyspark fails with python 3.6

Rising Star
Hi

Based on the error message you have shared.
...
TypeError: namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module'

This error corresponds to bug JIRA SPARK-19019 [1]. This bug relates to a compatibility issue between Spark and Python 3.6

Spark 1.6 requires Python 2.6+ as per the Document[2]

[1] https://issues.apache.org/jira/browse/SPARK-19019
[2] https://spark.apache.org/docs/1.6.0/#downloading
[3] https://www.cloudera.com/documentation/enterprise/5-14-x/topics/spark_python.html#spark_python__sect...

Re: pyspark fails with python 3.6

Explorer

Thanks, Agreed. I also found the bug details.

 

Based on the URL https://spark.apache.org/docs/1.6.0/#downloading you shared, it contains details which says it is compatible with 2.6+ and 3.1+ which is totally misleading since 3.6 is 3.1+

 

I have started working to upgrade my app to spark 2. Any suggestiosn on Spark 1.6 to Spark 2 migration guide on Cloudera cluster