The Compatibility Matrix is snazzy and nice, but it's all about OSes and browsers and such. There's no mention of Python.
We like Python3. JupyterHub uses Python3. Half of our Conda environments are Python3.
But HDP's PySpark does not like Python3. Especially scripts like
It's shebang line is
But if env python is Python3.x, there are print statements without brackets.
So what's the situation here? Is HDP 2.6.5 and 3.x compatible with Python3? Would it be a big effort to add the brackets to print statements in the PySpark scripts?
Thanks for the question. After doing some research I arrived at the conclusion that Hortonworks do not support Python 2.7.9 and above for any of the HDP/HDF stack versions as of now.
There is an enhancement request filed already under Hortonworks internal Jira. I shall update this question whenever I get a confirmation about Python 3.x support.
Hope it helps. Cheers!
Do we have any official notice from Cloudera/Hortonwroks on python 2.7 support has been taken off and we have to upgrade all our python versions to 3.x. with time lines.
We are running our pyspark jobs in python 3 conda environment and facing below WARNING logs and leads to more log size ~ 20 MB and difficult to trace the actual ERRORS. Is cloudera ready to support python 3 version (as no more support for python 2) on scripts like /etc/hadoop/conf/topology_script.py etc.
May I know solution for python 3 jobs to run in HDP 3.1.5.
20/03/10 18:26:01 WARN ScriptBasedMapping: Exception running /etc/hadoop/conf/topology_script.py 10.101.193.7
ExitCodeException exitCode=1: File "/etc/hadoop/conf/topology_script.py", line 63
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(rack)?