Looking for a supported version of Spark 3 for 7.3.1

Question

We’ve set up Cloudera Manager 7.13.1 along with CDP Private Cloud Base 7.3.1 on RHEL 9.4. As part of our environment validation, we’re looking to include Apache Spark 3 in the stack. However, we haven’t been able to locate a supported Spark 3 parcel that’s officially compatible with this combination.
We’ve reviewed the Cloudera Support Matrix, but couldn’t find a clear reference for Spark 3 support with these specific versions of CDP and CM.

According to a recent blog post, Announcing availability of Spark 3.5 on Cloudera Unified Runtime which also directs the reader to What's New in Apache Spark (7.3.1).

Apparently Spark 3 is supported by 7.3.1 and Spark 3.5 by 7.3.1 SP1.

Does Spark still get installed separately or is it included with the base runtime?

Should the support matrix be updated to reflect support for Spark 3?

jagadeesan · Accepted Answer

@moekraft Starting with Cloudera Runtime 7.3.x, Spark 3 is the default and integrated Spark version, and Spark 2 has been removed and is no longer supported.

>> Does Spark still get installed separately or is it included with the base runtime?
-- As a result, you do not need to install a separate Spark 3 parcel for CDP Private Cloud Base 7.3.x. The Spark 3 runtime is bundled within the Cloudera Runtime parcel itself so you won’t find a separate, compatible Spark 3 parcel in the support matrix or parcel repository for this version.

To proceed, simply use the Spark service that comes bundled with the Cloudera Runtime 7.3.x. After initial Cloudera Runtime 7.3.x installation, you can use the Add a Spark3 Service wizard to add and configure new service instances directly via Cloudera Manager

>> Apparently Spark 3 is supported by 7.3.1 and Spark 3.5 by 7.3.1 SP1.
- Cloudera Runtime 7.3.1.100 CHF 1 bundled with Spark 3.4.x. Please refer the list of the official component versions for Cloudera Runtime 7.3.1.100 CHF 1
- Cloudera Runtime 7.3.1.200 SP1 and latest bundled with Spark 3.5.x. Please refer the list of the official component versions for Cloudera Runtime 7.3.1.200 SP1.

>> Should the support matrix be updated to reflect support for Spark 3?
- It's for CDS version support matrix, from Cloudera Runtime 7.3.x onwards you don't need CDS for Spark3, so you can refer the below release notes:
https://docs.cloudera.com/cdp-private-cloud-base/7.3.1/private-release-notes/topics/rt-whats-new-spa...

DianaTorres · Answer

Hi @Bharati @jagadeesan @smdas do you have any insights here? Thanks!

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.