Python3.x Compatibility in HDP 2.6.x and 3.x

[#1]

The Compatibility Matrix is snazzy and nice, but it's all about OSes and browsers and such. There's no mention of Python.

We like Python3. JupyterHub uses Python3. Half of our Conda environments are Python3.

But HDP's PySpark does not like Python3. Especially scripts like

/usr/bin/hdp-select

It's shebang line is

#!/usr/bin/env python

But if env python is Python3.x, there are print statements without brackets.

So what's the situation here? Is HDP 2.6.5 and 3.x compatible with Python3? Would it be a big effort to add the brackets to print statements in the PySpark scripts?

[#2]

Hi Bruno,

Thanks for the question. After doing some research I arrived at the conclusion that Hortonworks do not support Python 2.7.9 and above for any of the HDP/HDF stack versions as of now.

There is an enhancement request filed already under Hortonworks internal Jira. I shall update this question whenever I get a confirmation about Python 3.x support.

Hope it helps. Cheers!

[#3]

May I remark that Python 2.7 will not be maintained past 2020?

https://pythonclock.org/

[#4]

Hi team, we are in 2020 and support for python2.7 has stopped.

[#5]

Hi Team,

Do we have any official notice from Cloudera/Hortonwroks on python 2.7 support has been taken off and we have to upgrade all our python versions to 3.x. with time lines.

[#6]

Hi All,

We are running our pyspark jobs in python 3 conda environment and facing below WARNING logs and leads to more log size ~ 20 MB and difficult to trace the actual ERRORS. Is cloudera ready to support python 3 version  (as no more support for python 2) on scripts like /etc/hadoop/conf/topology_script.py etc. 

May I know solution for python 3 jobs to run in HDP 3.1.5.

20/03/10 18:26:01 WARN ScriptBasedMapping: Exception running /etc/hadoop/conf/topology_script.py 10.101.193.7
ExitCodeException exitCode=1: File "/etc/hadoop/conf/topology_script.py", line 63
print rack
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(rack)?

at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)
at org.apache.hadoop.util.Shell.run(Shell.java:901)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
at