Installing spark3 (CDS 3.2)

[#1]

I have recently upgraded Private Cloud Base components (CM 7.5.4, CDH 7.1.7). I am unable to locate a Spark 3 or Spark3 LIVY service after following the CDS 3.2, installation steps. ( https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-install-spark-3-parcel.htm... )

I am able to download, distribute, and activate the Spark 3 parcel but there are no options to add it as a service. I did add the `csd` jars to the directories with no luck. logs indicate conflicts with the csd (I tried with both the page listed .49 and also the unlisted .1001). Removing the csd jars removes the warning but the service is still not available. 

"...

2022-02-11 12:26:19,206 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type SPARK3_ON_YARN. Latest version is 7.5.4

2022-02-11 12:26:19,208 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [SPARK3_ON_YARN-3.2.0.3.2.7170.1001-3], version 3.0.7110.0 ..."

2022-02-11 12:26:19,206 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type SPARK3_ON_YARN. Latest version is 7.5.4
2022-02-11 12:26:19,208 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [SPARK3_ON_YARN-3.2.0.3.2.7170.1001-3], version 3.0.7110.0
2022-02-11 12:26:19,209 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type LIVY_FOR_SPARK3. Latest version is 7.5.4
2022-02-11 12:26:19,209 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [LIVY_FOR_SPARK3-0.6.3000.3.2.7170.1001-3], version 3.0.7110.0
2022-02-11 12:26:21,980 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [LIVY_FOR_SPARK3-7.5.4]
2022-02-11 12:26:22,914 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK3_ON_YARN-7.5.4]
2022-02-11 12:26:22,927 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN53-7.5.4]
2022-02-11 12:26:22,939 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN54-7.5.4]
2022-02-11 12:26:22,950 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN5_10-7.5.4]
2022-02-11 12:26:22,963 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN5_11-7.5.4]
2022-02-11 12:26:22,974 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN6_0-7.5.4]
2022-02-11 12:26:22,986 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN6_2-7.5.4]
2022-02-11 12:26:22,998 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_0-7.5.4]
2022-02-11 12:26:23,010 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1-7.5.4]
2022-02-11 12:26:23,021 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1_5-7.5.4]
2022-02-11 12:26:23,033 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1_6-7.5.4]
2022-02-11 12:26:23,045 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1_7-7.5.4]

[#2]

1) Please download the SPARK3_ON_YARN-3.2.0.3.2.7170.0-49.jar from link below: 

https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-spark-3-packaging.html

2) copy the jar into /op/cloudera/csd on CM server host. Make sure there are no old CSDs in this directory. Delete all old ones.

3) Set the file ownership of the service descriptor to cloudera-scm:cloudera-scm with permission 644.

4) Restart the Cloudera Manager Server 

5) Make sure the parcel from the link below is downloaded, distributed, and activated

https://archive.cloudera.com/p/spark3/3.2.7170.0/parcels/

6) After all these steps then from CM > Cluster > Drop don menu > Add Service > Spark 3

[#3]

Still no `Spark 3` service after following the steps. Directory only contains LIVY and SPARK csd's. Server logs indicate that the csd is being ignored again.

$ ls -lah /opt/cloudera/csd

total 40K

drwxr-xr-x. 2 cloudera-scm cloudera-scm 102 Feb 16 14:52 .

drwxr-xr-x. 8 cloudera-scm cloudera-scm  97 Jan 12 21:57 ..

-rw-r--r--. 1 cloudera-scm cloudera-scm 17K Feb 16 14:52 LIVY_FOR_SPARK3-0.6.3000.3.2.7170.0-49.jar

-rw-r--r--. 1 cloudera-scm cloudera-scm 20K Feb 16 14:52 SPARK3_ON_YARN-3.2.0.3.2.7170.0-49.jar

Server logs:

2022-02-16 15:22:33,862 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type SPARK3_ON_YARN. Latest version is 7.5.4

2022-02-16 15:22:33,864 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [SPARK3_ON_YARN-3.2.0.3.2.7170.0-49], version 3.0.7110.0

2022-02-16 15:22:33,866 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type LIVY_FOR_SPARK3. Latest version is 7.5.4

2022-02-16 15:22:33,866 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [LIVY_FOR_SPARK3-0.6.3000.3.2.7170.0-49], version 3.0.7110.0

2022-02-16 15:22:36,433 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [LIVY_FOR_SPARK3-7.5.4]

2022-02-16 15:22:37,347 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK3_ON_YARN-7.5.4]

[#4 · Accepted solution]

Please follow the steps below:

• SSH to Cloudera Manager host where the Spark 3 CSDs are deployed
• Find the following files and use a file manager (for example mc) or an editor to open them as zip files and edit the contents of "descriptor/service.sdl". Probably the easiest way is to open the jar files with vim:
  • $ vim /opt/cloudera/csd/SPARK3_ON_YARN-3.2.0.3.2.7170.0-49.jar
  • $ vim /opt/cloudera/csd/LIVY_FOR_SPARK3-0.6.3000.3.2.7170.0-49.jar
• In the descriptor/service.sdl files, prefix the version with something that is higher than the CM version number, so instead of:
  • "version" : "3.0.7110.0", add "version" : "7.5.4.3.0.7110.0", 4.
• Restart CM server and wait until it comes back up:
  • $ service cloudera-scm-server restart
• Spark 3 can now be installed. Once installed, deploy client config and restart all services that have stale configs (most importantly YARN).

[#5]

That worked. Many thanks.

[#6]

@mhsyed Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks!