question Re: Installing spark3 (CDS 3.2) in Support Questions

Installing spark3 (CDS 3.2)

mhsyed — Tue, 21 Apr 2026 08:01:11 GMT

I have recently upgraded Private Cloud Base components (CM 7.5.4, CDH 7.1.7). I am unable to locate a Spark 3 or Spark3 LIVY service after following the CDS 3.2, installation steps. ( https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-install-spark-3-parcel.html )

I am able to download, distribute, and activate the Spark 3 parcel but there are no options to add it as a service. I did add the `csd` jars to the directories with no luck. logs indicate conflicts with the csd (I tried with both the page listed .49 and also the unlisted .1001). Removing the csd jars removes the warning but the service is still not available.

"...

2022-02-11 12:26:19,206 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type SPARK3_ON_YARN. Latest version is 7.5.4

2022-02-11 12:26:19,208 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [SPARK3_ON_YARN-3.2.0.3.2.7170.1001-3], version 3.0.7110.0 ..."

2022-02-11 12:26:19,206 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type SPARK3_ON_YARN. Latest version is 7.5.4 2022-02-11 12:26:19,208 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [SPARK3_ON_YARN-3.2.0.3.2.7170.1001-3], version 3.0.7110.0 2022-02-11 12:26:19,209 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type LIVY_FOR_SPARK3. Latest version is 7.5.4 2022-02-11 12:26:19,209 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [LIVY_FOR_SPARK3-0.6.3000.3.2.7170.1001-3], version 3.0.7110.0 2022-02-11 12:26:21,980 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [LIVY_FOR_SPARK3-7.5.4] 2022-02-11 12:26:22,914 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK3_ON_YARN-7.5.4] 2022-02-11 12:26:22,927 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN53-7.5.4] 2022-02-11 12:26:22,939 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN54-7.5.4] 2022-02-11 12:26:22,950 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN5_10-7.5.4] 2022-02-11 12:26:22,963 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN5_11-7.5.4] 2022-02-11 12:26:22,974 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN6_0-7.5.4] 2022-02-11 12:26:22,986 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN6_2-7.5.4] 2022-02-11 12:26:22,998 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_0-7.5.4] 2022-02-11 12:26:23,010 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1-7.5.4] 2022-02-11 12:26:23,021 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1_5-7.5.4] 2022-02-11 12:26:23,033 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1_6-7.5.4] 2022-02-11 12:26:23,045 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1_7-7.5.4]

Re: Installing spark3 (CDS 3.2)

Bharati — Fri, 11 Feb 2022 21:22:36 GMT

1) Please download the SPARK3_ON_YARN-3.2.0.3.2.7170.0-49.jar from link below:

https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-spark-3-packaging.html

2) copy the jar into /op/cloudera/csd on CM server host. Make sure there are no old CSDs in this directory. Delete all old ones.

3) Set the file ownership of the service descriptor to cloudera-scm:cloudera-scm with permission 644.

4) Restart the Cloudera Manager Server

5) Make sure the parcel from the link below is downloaded, distributed, and activated

https://archive.cloudera.com/p/spark3/3.2.7170.0/parcels/

6) After all these steps then from CM > Cluster > Drop don menu > Add Service > Spark 3

Re: Installing spark3 (CDS 3.2)

DianaTorres — Wed, 16 Feb 2022 14:37:55 GMT

@mhsyed Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks!

Re: Installing spark3 (CDS 3.2)

mhsyed — Wed, 16 Feb 2022 20:38:14 GMT

Still no `Spark 3` service after following the steps. Directory only contains LIVY and SPARK csd's. Server logs indicate that the csd is being ignored again.

$ ls -lah /opt/cloudera/csd

total 40K

drwxr-xr-x. 2 cloudera-scm cloudera-scm 102 Feb 16 14:52 .

drwxr-xr-x. 8 cloudera-scm cloudera-scm 97 Jan 12 21:57 ..

-rw-r--r--. 1 cloudera-scm cloudera-scm 17K Feb 16 14:52 LIVY_FOR_SPARK3-0.6.3000.3.2.7170.0-49.jar

-rw-r--r--. 1 cloudera-scm cloudera-scm 20K Feb 16 14:52 SPARK3_ON_YARN-3.2.0.3.2.7170.0-49.jar

Server logs:

2022-02-16 15:22:33,862 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type SPARK3_ON_YARN. Latest version is 7.5.4

2022-02-16 15:22:33,864 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [SPARK3_ON_YARN-3.2.0.3.2.7170.0-49], version 3.0.7110.0

2022-02-16 15:22:33,866 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type LIVY_FOR_SPARK3. Latest version is 7.5.4

2022-02-16 15:22:33,866 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [LIVY_FOR_SPARK3-0.6.3000.3.2.7170.0-49], version 3.0.7110.0

2022-02-16 15:22:36,433 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [LIVY_FOR_SPARK3-7.5.4]

2022-02-16 15:22:37,347 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK3_ON_YARN-7.5.4]

Re: Installing spark3 (CDS 3.2)

Bharati — Wed, 16 Feb 2022 23:18:19 GMT

Please follow the steps below:

SSH to Cloudera Manager host where the Spark 3 CSDs are deployed
Find the following files and use a file manager (for example mc) or an editor to open them as zip files and edit the contents of "descriptor/service.sdl". Probably the easiest way is to open the jar files with vim:
- $ vim /opt/cloudera/csd/SPARK3_ON_YARN-3.2.0.3.2.7170.0-49.jar
- $ vim /opt/cloudera/csd/LIVY_FOR_SPARK3-0.6.3000.3.2.7170.0-49.jar
In the descriptor/service.sdl files, prefix the version with something that is higher than the CM version number, so instead of:
- "version" : "3.0.7110.0", add "version" : "7.5.4.3.0.7110.0", 4.
Restart CM server and wait until it comes back up:
- $ service cloudera-scm-server restart
Spark 3 can now be installed. Once installed, deploy client config and restart all services that have stale configs (most importantly YARN).

Re: Installing spark3 (CDS 3.2)

mhsyed — Thu, 17 Feb 2022 15:53:27 GMT

That worked. Many thanks.