Member since
05-31-2017
34
Posts
8
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
861 | 05-09-2022 02:54 PM | |
739 | 04-28-2022 11:27 AM | |
422 | 04-19-2022 11:03 AM | |
1666 | 02-16-2022 03:18 PM |
10-19-2022
10:48 AM
1 Kudo
AFAIU It should not matter. You can choose the rolling restart option as well. So CM can decide the sequence of the broker restarts.
... View more
05-09-2022
02:54 PM
@bluespring One should not be deleting the offline/online partitions that may cause in data loss or under-replicated partitions. You may reassign the partitions to new hosts following the document below: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/kafka-managing/topics/kafka-manage-cli-reassign-overview.html
... View more
04-28-2022
01:55 PM
@clouderaskme The latest CDP 7.1.7 comes with the default Spark 2.4 version. https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/runtime-release-notes/topics/rt-pvc-runtime-component-versions.html Spark 2.4 supports Python 2.7 and 3.4-3.7. https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/release-guide/topics/cdpdc-os-requirements.html
... View more
04-28-2022
11:27 AM
@clouderaskme Please review the documents below that provides the details on requirements for Spark3.2, 3.1 and 3.0 https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-3-requirements.html https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/cds-3/topics/spark-spark-3-requirements.html https://docs.cloudera.com/cdp-private-cloud-base/7.1.4/cds-3/topics/spark-spark-3-requirements.html Cloudera Distributed Spark 3.2 requires Python 3.6+ and requires CDP 7.1.7 and higher Cloudera Distributed Spark 3.1 requires Python 3.6+ and requires CDP 7.1.7 and higher Cloudera Distributed Spark 3.0 requires Python 3.4 or higher. and requires CDP7.1.3, 7.1.4 and 7.1.5
... View more
04-19-2022
11:03 AM
1 Kudo
@Sayed016 Thank you for your question. From the error stack of CM logs, it looks like it tries to copy the system database and information_schema. You need to exclude the system database and information_schema Add the following exclusion on a Hive replication: Databases: (?!information_schema|sys\b).+ Tables: [\w].+
... View more
02-16-2022
03:18 PM
2 Kudos
Please follow the steps below: SSH to Cloudera Manager host where the Spark 3 CSDs are deployed Find the following files and use a file manager (for example mc) or an editor to open them as zip files and edit the contents of "descriptor/service.sdl". Probably the easiest way is to open the jar files with vim: $ vim /opt/cloudera/csd/ SPARK3_ON_YARN-3.2.0.3.2.7170.0-49.jar $ vim /opt/cloudera/csd/ LIVY_FOR_SPARK3-0.6.3000.3.2.7170.0-49.jar In the descriptor/service.sdl files, prefix the version with something that is higher than the CM version number, so instead of: "version" : "3.0.7110.0", add "version" : "7.5.4.3.0.7110.0", 4. Restart CM server and wait until it comes back up: $ service cloudera-scm-server restart Spark 3 can now be installed. Once installed, deploy client config and restart all services that have stale configs (most importantly YARN).
... View more
02-11-2022
01:21 PM
1 Kudo
1) Please download the SPARK3_ON_YARN-3.2.0.3.2.7170.0-49.jar from link below: https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-spark-3-packaging.html 2) copy the jar into /op/cloudera/csd on CM server host. Make sure there are no old CSDs in this directory. Delete all old ones. 3) Set the file ownership of the service descriptor to cloudera-scm:cloudera-scm with permission 644. 4) R estart the Cloudera Manager Server 5) Make sure the parcel from the link below is downloaded, distributed, and activated https://archive.cloudera.com/p/spark3/3.2.7170.0/parcels/ 6) After all these steps then from CM > Cluster > Drop don menu > Add Service > Spark 3
... View more
01-09-2020
11:20 AM
1. CDH5 works with Spark2 by installing a separate parcel. 2. The document suggests that CDH5 + Spark2 does not support Spark-On-HBase. Spark-On-HBase work with CDH 5 and Spark 1.6 3. With CDH 6.x default Spark version is Spark 2.x+ so as a new feature Spark-On-HBase https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_600_new_features.html#spark_new_features Spark On HBase now runs on top of Apache Spark 2.x. 4. Refer document below to use Spark-on-HBase on CDH6.x: https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_hbase_import.html#concept_asc_ctz_wp
... View more
01-11-2019
02:45 PM
Please attach the clouderaserver and cloudera scm agent logs to further look into the issue along with screenshot of CM home page.
... View more
01-11-2019
02:13 PM
1) The error that you showed talks about SPARK2_ON_YARN-2.3.0.cloudera2.jar not SPARK2_ON_YARN-2.3.0.cloudera4.jar. These are 2 different versions. ERROR 1704117900@scm-web-7:com.cloudera.csd.components.C sdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3. 0.cloudera2.jar, Error: CSD bundle data directory or Jar file doesn't exist 2) Do you multiple CSDs in /opt/clodera/csd or parcels activated? If that's the case please deactovate other versions and delete all old CSD. Make sure the parcel and the CSD jar version matches.
... View more
01-11-2019
01:36 PM
I see that you are on CDH 5.16. Spark 2.3 is compatible with that. What is the CM version you are on? - Can you confirm if you have installed Spark 2.3 parcel in CM > Hosts > Parcels? Send us the screenshot of the entire page if possible. - Also please post the directory permission for following: ls -ltr /opt/cloudera/ ls -ltr /opt/cloudera/csd
... View more
01-11-2019
01:28 PM
I see that you are on CDH 5.16. Spark 2.3 is compatible with that. What is the CM version you are on? - Can you confirm if you have installed Spark 2.3 parcel in CM > Hosts > Parcels? - Please post the directory permission for following: ls -ltr /opt/cloudera/ ls -ltr /opt/cloudera/csd
... View more
01-07-2019
04:23 PM
I think you are missing the parcel installation along with CDS download. Whenever upgrading spark 2.x versions please follow all steps below and make sure there aren't any old version CSD in CSD folder
A) Make sure the Parcel is installed distributed and Activated. Refer Step 3 in the document below:
https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html
B) Make sure you download Spark 2 CDS on Cloudera Manager Server host and delete old CDS
https://www.cloudera.com/documentation/spark2/latest/topics/spark2_packaging.html#versions
C) and followed steps mentioned in the link below
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_addon_services.html#concept_kpt_spj_bn
- Installing the Custom Service Descriptor File
- Acquire the CSD file from Cloudera or an ISV.
- Log on to the Cloudera Manager Server host, and place the CSD file under the location configured for CSD files.
- Set the file ownership to cloudera-scm:cloudera-scm with permission 644.
- Restart the Cloudera Manager Server:
service cloudera-scm-server restart
- Log into the Cloudera Manager Admin Console and restart the Cloudera Management Service.
Do one of the following:
a)
Select Clusters > Cloudera Management Service.
Select Actions > Restart.
b)
On the Home > Status tab, click to the right of Cloudera Management Service and select Restart.
- Click Restart to confirm. The Command Details window shows the progress of stopping and then starting the roles.
- When Command completed with n/n successful subcommands appears, the task is complete. Click Close.
- Then try to Add Service for Spark 2.
... View more
05-31-2017
12:09 PM
3 Kudos
1) Make sure the Parcel is installed distributed and Activated. Refer Step 3 in the document below: https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html 2) Have you downloaded the Spark 2 CSD on Cloudera Manager Server host and followed steps mentioned in the link below? https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_addon_services.html#concept_kpt_spj_bn - Installing the Custom Service Descriptor File - Acquire the CSD file from Cloudera or an ISV. - Log on to the Cloudera Manager Server host, and place the CSD file under the location configured for CSD files. - Set the file ownership to cloudera-scm:cloudera-scm with permission 644. - Restart the Cloudera Manager Server: service cloudera-scm-server restart - Log into the Cloudera Manager Admin Console and restart the Cloudera Management Service. Do one of the following: a) Select Clusters > Cloudera Management Service. Select Actions > Restart. b) On the Home > Status tab, click to the right of Cloudera Management Service and select Restart. - Click Restart to confirm. The Command Details window shows the progress of stopping and then starting the roles. - When Command completed with n/n successful subcommands appears, the task is complete. Click Close. - Then try to Add Service for Spark 2.
... View more