Created 04-28-2022 10:47 AM
hello all,
I am currently running a python version of 3.6.8, Can I still install spark 3 on my cdp cluster?
Created on 04-28-2022 11:27 AM - edited 04-28-2022 11:31 AM
@clouderaskme Please review the documents below that provides the details on requirements for Spark3.2, 3.1 and 3.0
https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-3-requirements.html
https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/cds-3/topics/spark-spark-3-requirements.html
https://docs.cloudera.com/cdp-private-cloud-base/7.1.4/cds-3/topics/spark-spark-3-requirements.html
Cloudera Distributed Spark 3.2 requires Python 3.6+ and requires CDP 7.1.7 and higher
Cloudera Distributed Spark 3.1 requires Python 3.6+ and requires CDP 7.1.7 and higher
Cloudera Distributed Spark 3.0 requires Python 3.4 or higher. and requires CDP7.1.3, 7.1.4 and 7.1.5
Created on 04-28-2022 11:27 AM - edited 04-28-2022 11:31 AM
@clouderaskme Please review the documents below that provides the details on requirements for Spark3.2, 3.1 and 3.0
https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-3-requirements.html
https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/cds-3/topics/spark-spark-3-requirements.html
https://docs.cloudera.com/cdp-private-cloud-base/7.1.4/cds-3/topics/spark-spark-3-requirements.html
Cloudera Distributed Spark 3.2 requires Python 3.6+ and requires CDP 7.1.7 and higher
Cloudera Distributed Spark 3.1 requires Python 3.6+ and requires CDP 7.1.7 and higher
Cloudera Distributed Spark 3.0 requires Python 3.4 or higher. and requires CDP7.1.3, 7.1.4 and 7.1.5
Created 04-28-2022 11:46 AM
but If i want to continue having spark 2 on it, until what version can python updated?
Created 04-28-2022 01:55 PM
@clouderaskme The latest CDP 7.1.7 comes with the default Spark 2.4 version.
Spark 2.4 supports Python 2.7 and 3.4-3.7.