AFAIU It should not matter. You can choose the rolling restart option as well. So CM can decide the sequence of the broker restarts.  ... View more

@clouderaskme  The latest CDP 7.1.7 comes with the default Spark 2.4 version.  https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/runtime-release-notes/topics/rt-pvc-runtime-component-versions.html Spark 2.4 supports Python 2.7 and 3.4-3.7. https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/release-guide/topics/cdpdc-os-requirements.html   ... View more

1) Please download the SPARK3_ON_YARN-3.2.0.3.2.7170.0-49.jar from link below:  https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-spark-3-packaging.html 2) copy the jar into /op/cloudera/csd on CM server host. Make sure there are no old CSDs in this directory. Delete all old ones. 3) Set the file ownership of the service descriptor to   cloudera-scm:cloudera-scm   with permission 644. 4) R estart the Cloudera Manager Server  5) Make sure the parcel from the link below is downloaded, distributed, and activated https://archive.cloudera.com/p/spark3/3.2.7170.0/parcels/ 6) After all these steps then from CM > Cluster > Drop don menu > Add Service > Spark 3 ... View more

1. CDH5 works with Spark2 by installing a separate parcel. 2. The document suggests that CDH5 + Spark2 does not support Spark-On-HBase. Spark-On-HBase work with CDH 5 and Spark 1.6 3. With CDH 6.x default Spark version is Spark 2.x+ so as a new feature Spark-On-HBase https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_600_new_features.html#spark_new_features Spark On HBase now runs on top of Apache Spark 2.x. 4. Refer document below to use Spark-on-HBase on CDH6.x: https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_hbase_import.html#concept_asc_ctz_wp     ... View more

Please attach the clouderaserver and cloudera scm agent logs to further look into the issue along with screenshot of CM home page. ... View more

1) The error that you showed talks about SPARK2_ON_YARN-2.3.0.cloudera2.jar not SPARK2_ON_YARN-2.3.0.cloudera4.jar. These are 2 different versions. ERROR 1704117900@scm-web-7:com.cloudera.csd.components.C                                                                                        sdLocalRepository: Failed to load CSD from /opt/cloudera/csd/SPARK2_ON_YARN-2.3.                                                                                        0.cloudera2.jar, Error: CSD bundle data directory or Jar file doesn't exist   2) Do you multiple CSDs in /opt/clodera/csd or parcels activated? If that's the case please deactovate other versions and delete all old CSD. Make sure the parcel and the CSD jar version matches.     ... View more

I see that you are on CDH 5.16. Spark 2.3 is compatible with that. What is the CM version you are on?   - Can you confirm if you have installed Spark 2.3 parcel in CM > Hosts > Parcels? Send us the screenshot of the entire page if possible.   - Also please post the directory permission for following: ls -ltr /opt/cloudera/   ls -ltr /opt/cloudera/csd ... View more

I see that you are on CDH 5.16. Spark 2.3 is compatible with that. What is the CM version you are on?   - Can you confirm if you have installed Spark 2.3 parcel in CM > Hosts > Parcels? - Please post the directory permission for following: ls -ltr /opt/cloudera/   ls -ltr /opt/cloudera/csd         ... View more

I think you are missing the parcel installation along with CDS download. Whenever upgrading spark 2.x versions please follow all steps below and make sure there aren't any old version CSD in CSD folder A) Make sure the Parcel is installed distributed and Activated. Refer Step 3 in the document below: https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html B) Make sure you download Spark 2 CDS on Cloudera Manager Server host and delete old CDS https://www.cloudera.com/documentation/spark2/latest/topics/spark2_packaging.html#versions C) and followed steps mentioned in the link below https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_addon_services.html#concept_kpt_spj_bn - Installing the Custom Service Descriptor File - Acquire the CSD file from Cloudera or an ISV. - Log on to the Cloudera Manager Server host, and place the CSD file under the location configured for CSD files. - Set the file ownership to cloudera-scm:cloudera-scm with permission 644. - Restart the Cloudera Manager Server: service cloudera-scm-server restart - Log into the Cloudera Manager Admin Console and restart the Cloudera Management Service. Do one of the following: a) Select Clusters > Cloudera Management Service. Select Actions > Restart. b) On the Home > Status tab, click to the right of Cloudera Management Service and select Restart. - Click Restart to confirm. The Command Details window shows the progress of stopping and then starting the roles. - When Command completed with n/n successful subcommands appears, the task is complete. Click Close. - Then try to Add Service for Spark 2. ... View more

1)  Make sure the Parcel is installed distributed and Activated. Refer Step 3 in the document below: https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html 2) Have you downloaded the Spark 2 CSD on Cloudera Manager Server host and followed steps mentioned in the link below? https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_addon_services.html#concept_kpt_spj_bn - Installing the Custom Service Descriptor File - Acquire the CSD file from Cloudera or an ISV. - Log on to the Cloudera Manager Server host, and place the CSD file under the location configured for CSD files. - Set the file ownership to cloudera-scm:cloudera-scm with permission 644. - Restart the Cloudera Manager Server: service cloudera-scm-server restart - Log into the Cloudera Manager Admin Console and restart the Cloudera Management Service. Do one of the following: a) Select Clusters > Cloudera Management Service. Select Actions > Restart. b) On the Home > Status tab, click to the right of Cloudera Management Service and select Restart. - Click Restart to confirm. The Command Details window shows the progress of stopping and then starting the roles. - When Command completed with n/n successful subcommands appears, the task is complete. Click Close. - Then try to Add Service for Spark 2. ... View more