Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Python3.x Compatibility in HDP 2.6.x and 3.x

Highlighted

Python3.x Compatibility in HDP 2.6.x and 3.x

New Contributor

The Compatibility Matrix is snazzy and nice, but it's all about OSes and browsers and such. There's no mention of Python.

We like Python3. JupyterHub uses Python3. Half of our Conda environments are Python3.

But HDP's PySpark does not like Python3. Especially scripts like

/usr/bin/hdp-select

It's shebang line is

 

#!/usr/bin/env python

But if env python is Python3.x, there are print statements without brackets.

So what's the situation here? Is HDP 2.6.5 and 3.x compatible with Python3? Would it be a big effort to add the brackets to print statements in the PySpark scripts?

5 REPLIES 5
Highlighted

Re: Python3.x Compatibility in HDP 2.6.x and 3.x

Cloudera Employee

Hi Bruno,

Thanks for the question. After doing some research I arrived at the conclusion that Hortonworks do not support Python 2.7.9 and above for any of the HDP/HDF stack versions as of now.

There is an enhancement request filed already under Hortonworks internal Jira. I shall update this question whenever I get a confirmation about Python 3.x support.

Hope it helps. Cheers!

Highlighted

Re: Python3.x Compatibility in HDP 2.6.x and 3.x

New Contributor

May I remark that Python 2.7 will not be maintained past 2020?

https://pythonclock.org/

Highlighted

Re: Python3.x Compatibility in HDP 2.6.x and 3.x

New Contributor

Hi team, we are in 2020 and support for python2.7 has stopped.

Highlighted

Re: Python3.x Compatibility in HDP 2.6.x and 3.x

New Contributor

Hi Team,

Do we have any official notice from Cloudera/Hortonwroks on python 2.7 support has been taken off and we have to upgrade all our python versions to 3.x. with time lines.


Highlighted

Re: Python3.x Compatibility in HDP 2.6.x and 3.x

New Contributor

Hi All,

 

We are running our pyspark jobs in python 3 conda environment and facing below WARNING logs and leads to more log size ~ 20 MB and difficult to trace the actual ERRORS. Is cloudera ready to support python 3 version  (as no more support for python 2) on scripts like /etc/hadoop/conf/topology_script.py etc. 

 

May I know solution for python 3 jobs to run in HDP 3.1.5.

 

20/03/10 18:26:01 WARN ScriptBasedMapping: Exception running /etc/hadoop/conf/topology_script.py 10.101.193.7
ExitCodeException exitCode=1: File "/etc/hadoop/conf/topology_script.py", line 63
print rack
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(rack)?

at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)
at org.apache.hadoop.util.Shell.run(Shell.java:901)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
at

 

Don't have an account?
Coming from Hortonworks? Activate your account here