Silver ⭐️

Solved

spark 2 and spark 3 on cdp

3 replies
5585 views
Translate All

hello all,

I am currently running a python version of 3.6.8, Can I still install spark 3 on my cdp cluster?

Gold ⭐️

@clouderaskme Please review the documents below that provides the details on requirements for Spark3.2, 3.1 and 3.0
https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-3-requirements.html

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/cds-3/topics/spark-spark-3-requirements.html

https://docs.cloudera.com/cdp-private-cloud-base/7.1.4/cds-3/topics/spark-spark-3-requirements.html

Cloudera Distributed Spark 3.2 requires Python 3.6+ and requires CDP 7.1.7 and higher
Cloudera Distributed Spark 3.1 requires Python 3.6+ and requires CDP 7.1.7 and higher
Cloudera Distributed Spark 3.0 requires Python 3.4 or higher. and requires CDP7.1.3, 7.1.4 and 7.1.5

Gold ⭐️

@clouderaskme Please review the documents below that provides the details on requirements for Spark3.2, 3.1 and 3.0
https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-3-requirements.html

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/cds-3/topics/spark-spark-3-requirements.html

https://docs.cloudera.com/cdp-private-cloud-base/7.1.4/cds-3/topics/spark-spark-3-requirements.html

Cloudera Distributed Spark 3.2 requires Python 3.6+ and requires CDP 7.1.7 and higher
Cloudera Distributed Spark 3.1 requires Python 3.6+ and requires CDP 7.1.7 and higher
Cloudera Distributed Spark 3.0 requires Python 3.4 or higher. and requires CDP7.1.3, 7.1.4 and 7.1.5

Translate

clouderaskmeAuthor

Silver ⭐️

but If i want to continue having spark 2 on it, until what version can python updated?

Translate

Gold ⭐️

@clouderaskme The latest CDP 7.1.7 comes with the default Spark 2.4 version.

https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/runtime-release-notes/topics/rt-pvc-runtime-component-versions.html

Spark 2.4 supports Python 2.7 and 3.4-3.7.

https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/release-guide/topics/cdpdc-os-requirements.html

Translate