Hi @shraddha    Could you please check by any chance if you have set master as local while creating SparkSession in your code.   Use the following sample code to run locally and cluster without updating the master value.  val appName = "MySparkApp" // Creating the SparkConf object val sparkConf = new SparkConf().setAppName(appName).setIfMissing("spark.master", "local[2]") // Creating the SparkSession object val spark: SparkSession = SparkSession.builder().config(sparkConf).getOrCreate()   Verify the whole logs once again to check is there any others errors. ... View more

Hi @Yosieam    Thanks for sharing the code. You forgot to share the spark-submit/pyspark command. Please check what is executor/driver memory is passed to the spark-submit. Could you please confirm file is in local system/hdfs system.   ... View more

Hi @dmharshit    As you know, Cloudera provides the Hybrid data platform, so you can install the CDP product in on-premises and public cloud or both.   CDP Private Cloud Base product is supported only for On-Premises cluster.  CDP Public Cloud Base product is supported for public cloud like AWS, Azure, GCP.   @fzsombor already shared references how you can install CDP Private cloud and how to install Spark3 as well.    Please let me know still you need any further information.     ... View more

Hi @Camilo    When you are sharing the exception you need to share more details. So it will help us to provide a solution in faster way.   1. How are you launching the spark job? 2. If you built application using maven or sbt built tool have you specified spark-hive.jar version. For example, <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hive_2.12</artifactId> <version>2.4.8</version> <scope>provided</scope> </dependency>   References:   1. https://stackoverflow.com/questions/39444493/how-to-create-sparksession-with-hive-support-fails-with-hive-classes-are-not-f 2. https://mvnrepository.com/artifact/org.apache.spark/spark-hive   ... View more

What is the HDP version. if it is HDP3.x then you need to use Hive Warehouse Connector (HWC). ... View more

Hi @mala_etl    You can find the catalog information in the below link:   https://stackoverflow.com/questions/59894454/spark-and-hive-in-hadoop-3-difference-between-metastore-catalog-default-and-spa   Could you please confirm, the table is internal or external table in Hive and also verify the data in Hive. ... View more

Hi @mala_etl    I think you didn't mention you are running the application in CDH/HDP/CDP. Could you please share your hive script and check you are using hive catalog instead of in-memory catalog.   ... View more

Hi @somant    Please don't use open source libraries and use cluster-supported spark/kafka versions.    Check the following example code:   https://community.cloudera.com/t5/Community-Articles/Running-DirectKafkaWordCount-example-in-CDP/ta-p/340402 ... View more