About mhsyed

mhsyed · ‎08-04-2022

We are trying to determine the correct latest version of Cloudera Runtime to use, CM is offering 2 possibilities for CDH 7.1.7-1: 7.1.7-1.cdh7.1.7.p74.21057765 7.1.7-1.cdh7.1.7.p1000.24102687 What is the best way to determine the latest version? Thanks in advance.

mhsyed · ‎03-10-2022

I am unable to add the `mkl-wrapper` parcel. The repo is no longer available. Since my use case is for Spark, according to the instructions ,I need the JNI wrapper for spark to use the MKL BLAS and LAPACK. Broken link: https://raw.githubusercontent.com/Intel-bigdata/mkl-wrappers-parcel-repo/master/ Thanks in advance.

mhsyed · ‎02-17-2022

That worked. Many thanks.

mhsyed · ‎02-16-2022

Still no `Spark 3` service after following the steps. Directory only contains LIVY and SPARK csd's. Server logs indicate that the csd is being ignored again. $ ls -lah /opt/cloudera/csd total 40K drwxr-xr-x. 2 cloudera-scm cloudera-scm 102 Feb 16 14:52 . drwxr-xr-x. 8 cloudera-scm cloudera-scm 97 Jan 12 21:57 .. -rw-r--r--. 1 cloudera-scm cloudera-scm 17K Feb 16 14:52 LIVY_FOR_SPARK3-0.6.3000.3.2.7170.0-49.jar -rw-r--r--. 1 cloudera-scm cloudera-scm 20K Feb 16 14:52 SPARK3_ON_YARN-3.2.0.3.2.7170.0-49.jar Server logs: 2022-02-16 15:22:33,862 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type SPARK3_ON_YARN. Latest version is 7.5.4 2022-02-16 15:22:33,864 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [SPARK3_ON_YARN-3.2.0.3.2.7170.0-49], version 3.0.7110.0 2022-02-16 15:22:33,866 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type LIVY_FOR_SPARK3. Latest version is 7.5.4 2022-02-16 15:22:33,866 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [LIVY_FOR_SPARK3-0.6.3000.3.2.7170.0-49], version 3.0.7110.0 2022-02-16 15:22:36,433 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [LIVY_FOR_SPARK3-7.5.4] 2022-02-16 15:22:37,347 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK3_ON_YARN-7.5.4]

mhsyed · ‎02-11-2022

I have recently upgraded Private Cloud Base components (CM 7.5.4, CDH 7.1.7). I am unable to locate a Spark 3 or Spark3 LIVY service after following the CDS 3.2, installation steps. ( https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-install-spark-3-parcel.html ) I am able to download, distribute, and activate the Spark 3 parcel but there are no options to add it as a service. I did add the `csd` jars to the directories with no luck. logs indicate conflicts with the csd (I tried with both the page listed .49 and also the unlisted .1001). Removing the csd jars removes the warning but the service is still not available. "... 2022-02-11 12:26:19,206 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type SPARK3_ON_YARN. Latest version is 7.5.4 2022-02-11 12:26:19,208 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [SPARK3_ON_YARN-3.2.0.3.2.7170.1001-3], version 3.0.7110.0 ..." 2022-02-11 12:26:19,206 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type SPARK3_ON_YARN. Latest version is 7.5.4 2022-02-11 12:26:19,208 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [SPARK3_ON_YARN-3.2.0.3.2.7170.1001-3], version 3.0.7110.0 2022-02-11 12:26:19,209 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type LIVY_FOR_SPARK3. Latest version is 7.5.4 2022-02-11 12:26:19,209 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [LIVY_FOR_SPARK3-0.6.3000.3.2.7170.1001-3], version 3.0.7110.0 2022-02-11 12:26:21,980 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [LIVY_FOR_SPARK3-7.5.4] 2022-02-11 12:26:22,914 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK3_ON_YARN-7.5.4] 2022-02-11 12:26:22,927 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN53-7.5.4] 2022-02-11 12:26:22,939 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN54-7.5.4] 2022-02-11 12:26:22,950 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN5_10-7.5.4] 2022-02-11 12:26:22,963 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN5_11-7.5.4] 2022-02-11 12:26:22,974 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN6_0-7.5.4] 2022-02-11 12:26:22,986 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN6_2-7.5.4] 2022-02-11 12:26:22,998 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_0-7.5.4] 2022-02-11 12:26:23,010 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1-7.5.4] 2022-02-11 12:26:23,021 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1_5-7.5.4] 2022-02-11 12:26:23,033 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1_6-7.5.4] 2022-02-11 12:26:23,045 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN7_1_7-7.5.4]

Online	Offline
Last Visited	‎09-06-2022 04:05 PM

Member Since	‎02-11-2022 10:05 AM
Last Visited	‎09-06-2022 04:05 PM
Posts	5

Cloudera Community

Determine Latest CDH (7.1.7-1.cdh7.1.7{.p74.210577...

404 when Adding MKL wrapper parcel

Re: Installing spark3 (CDS 3.2)

Re: Installing spark3 (CDS 3.2)

Installing spark3 (CDS 3.2)