Support Questions

Find answers, ask questions, and share your expertise

python 3.6 or higher installation on CDP

avatar
New Contributor

Hi

We are deploying on premise CDP 7.3 with CDSW 1.8 over RHEL 7.8 which comes with python 2.6 as default. Now we are planning to upgrade python to 3.6. Is there any specific steps I need to take in cluster config beside yum installation. 

 

regards

 

 

1 ACCEPTED SOLUTION

avatar
Master Guru

@Mamun_Shaheed CDP doesn’t support Python 3 and higher for CDH services. Here is the Software Dependency Note for reference: 

Python - CDP Private Cloud Base, with the exceptions of Hue and Spark, is supported on the Python version that is included in the operating system by default, as well as higher versions, but is not compatible with Python 3.0 or higher.

For example, CDP Private Cloud Base requires Python 2.7 or higher on RHEL 7 compatible operating systems.

Spark 2 requires Python 2.7 or higher, and supports Python 3. If the right level of Python is not picked up by default, set the PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables to point to the correct Python executable before running the pyspark command.

 I am assuming you want to use Python 3 for CDSW etc. For that you can use custom engine with required Python version which is independent with CDH services. Some reference docs are below. 

https://docs.cloudera.com/documentation/data-science-workbench/1-8-x/topics/cdsw_extensible_engines....

 

In short you can use the distinct Python in hosts but just make sure Cloudera services are using only the supported Python version. 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

1 REPLY 1

avatar
Master Guru

@Mamun_Shaheed CDP doesn’t support Python 3 and higher for CDH services. Here is the Software Dependency Note for reference: 

Python - CDP Private Cloud Base, with the exceptions of Hue and Spark, is supported on the Python version that is included in the operating system by default, as well as higher versions, but is not compatible with Python 3.0 or higher.

For example, CDP Private Cloud Base requires Python 2.7 or higher on RHEL 7 compatible operating systems.

Spark 2 requires Python 2.7 or higher, and supports Python 3. If the right level of Python is not picked up by default, set the PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables to point to the correct Python executable before running the pyspark command.

 I am assuming you want to use Python 3 for CDSW etc. For that you can use custom engine with required Python version which is independent with CDH services. Some reference docs are below. 

https://docs.cloudera.com/documentation/data-science-workbench/1-8-x/topics/cdsw_extensible_engines....

 

In short you can use the distinct Python in hosts but just make sure Cloudera services are using only the supported Python version. 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.