Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

topology.py not Python 3 compatible

avatar
New Contributor

I don't know who's responsible for writing topology.py, but it uses Python 2 syntax, so if I try to run PySpark with Python 3 using

 

export PYSPARK_PYTHON=python3

 

I get tons of stacktraces.

 

Uri

2 ACCEPTED SOLUTIONS

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
5 REPLIES 5

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
New Contributor

Any workaround for earlier versions?  I'm on 5.5 and I don't manage the cluster.

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer

Hi,

     I'm cluster manager,and CDH version is 5.7.2.

      The same trouble,If I can change some params in CM to solve this problem. 

avatar
New Contributor

One easy option in old CDH versions is just to change the shebang at the beginning of the script to:

 

#!/usr/bin/env python2

Actually this will be more precise than the default, because the script is actually a python 2 script and ... python 3 is coming ... so it is good to be specific about the given version of the interpreter the script needs. We could be even more explicit and require python2.7.