Member since
11-12-2018
218
Posts
178
Kudos Received
35
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
273 | 08-08-2025 04:22 PM | |
342 | 07-11-2025 08:48 PM | |
542 | 07-09-2025 09:33 PM | |
1081 | 04-26-2024 02:20 AM | |
1427 | 04-18-2024 12:35 PM |
08-08-2025
04:22 PM
@Malrashed, If you are using Cloudera Runtime 7.1.9 then you can use either CDS 3.3 or CDS 3.5 Powered by Apache Spark as an add-on service. For more details, you can refer to this document. Please note, CDS 3.3 Powered by Apache Spark 3.3.x and CDS 3.5 Powered by Apache Spark 3.5.x are distributed as a parcel (Refer here for additional download details). There are no external Custom Service Descriptors (CSD) for Livy for Spark3 or Spark3 because they are already part of Cloudera Manager 7.11.3. In Cloudera Runtime 7.1.9, Spark 2 is the default. If you need to use Spark 3, it must be added as an add-on service. Note that Spark 2 is deprecated in Cloudera Runtime 7.1.9. Starting with Cloudera Runtime 7.3.x, Spark 3 becomes the default version
... View more
07-14-2025
06:27 AM
1 Kudo
Thank you for confirming!
... View more
07-11-2025
08:48 PM
@moekraft Starting with Cloudera Runtime 7.3.x, Spark 3 is the default and integrated Spark version, and Spark 2 has been removed and is no longer supported. >> Does Spark still get installed separately or is it included with the base runtime? -- As a result, you do not need to install a separate Spark 3 parcel for CDP Private Cloud Base 7.3.x. The Spark 3 runtime is bundled within the Cloudera Runtime parcel itself so you won’t find a separate, compatible Spark 3 parcel in the support matrix or parcel repository for this version. To proceed, simply use the Spark service that comes bundled with the Cloudera Runtime 7.3.x. After initial Cloudera Runtime 7.3.x installation, you can use the Add a Spark3 Service wizard to add and configure new service instances directly via Cloudera Manager >> Apparently Spark 3 is supported by 7.3.1 and Spark 3.5 by 7.3.1 SP1. - Cloudera Runtime 7.3.1.100 CHF 1 bundled with Spark 3.4.x. Please refer the list of the official component versions for Cloudera Runtime 7.3.1.100 CHF 1 - Cloudera Runtime 7.3.1.200 SP1 and latest bundled with Spark 3.5.x. Please refer the list of the official component versions for Cloudera Runtime 7.3.1.200 SP1. >> Should the support matrix be updated to reflect support for Spark 3? - It's for CDS version support matrix, from Cloudera Runtime 7.3.x onwards you don't need CDS for Spark3, so you can refer the below release notes: https://docs.cloudera.com/cdp-private-cloud-base/7.3.1/private-release-notes/topics/rt-whats-new-spa...
... View more
03-03-2025
01:24 AM
Writing the output back into the database results in the same error. However, I checked the physical and logical plan of these operations and noticed that Spark does a "relation" operation for the second table that is over 900GB, reading all the columns within it instead of choosing the subset that is in the query. Thus, I translated the whole code into SQL and returned the table in dataframe format perfectly... Perhaps you have an idea why doesn't Spark push down the filtering and column prunning operations?
... View more
04-26-2024
02:20 AM
1 Kudo
Flume, Storm, Druid, Falcon, Mahout, Ambari, Pig, Sentry, and Navigator have changed or been removed in CDP with replaced components . For Storm can be replaced with Cloudera Streaming Analytics (CSA) powered by Apache Flink. Contact your Cloudera account team for more information about moving from Storm to CSA. You can refer comparing Storm and Flink also Migrating from Storm to Flink.
... View more
04-22-2024
12:46 PM
@Alaaeldin Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
... View more
04-24-2023
07:25 AM
Sorry for late response, I use oozie to submit a spark job
... View more
12-02-2022
09:44 AM
@sss123 Are you able to run spark commands via spark-shell spark-submit?
... View more
12-01-2022
02:25 AM
Hello @QiDam As stated by JD above, the CDE Service relies on Ozone Service on Base Cluster. If Ozone Service wasn't in Healthy State, the CDE Service enabling would fail with similar tracing as shared by you. We would recommend the following checks: Ensure Ozone Service is Up & Running on Base Cluster. Create a new Environment & check CDE Service enabling on the new Environment. If the above CDE Service enabling on new Environment is Successful, Reattempt the CDE Service enabling on the existing Environment. If the above Suggestion doesn't help, We would suggest engaging Support as any further troubleshooting would require sharing the Logs over the Public Community forum, which may have Customer's details. We shall mark the Post as Resolved now. If you have any concerns, Feel free to Update the Post likewise. Regards, Smarak
... View more
11-01-2022
12:01 AM
Hi @Siddu198 Add this config to your job: set("mapreduce.fileoutputcommitter.algorithm.version","2")
... View more