Hi @Ploeplse  Could you please share reproducible sample code and impala tab creation script? ... View more

Hi @mmk    I think you have shared the following information.   7 nodes with each having 250 gb memory and vcpu = 32 per each node   spark-defaults.conf  spark.executor.memory = 100g spark.executor.memoryOverhead = 49g spark.driver.memoryOverhead=200g spark.driver.memory = 500g   You have maximum of 250 gb for node and you have specified driver memory is (500gb and 200gb overhead). How it possible to driver to get 700gb?   Generally you should not exceed the driver/executor memory beyond yarn physical memory.   Coming to the actual problem, please avoid the show() to print 8000000 records. If you need to get the print the all values, then implement a logic to 1000 records at once and next 1000 records for another iteration.    https://stackoverflow.com/questions/29227949/how-to-implement-spark-sql-pagination-query   ... View more

Hi @mmk    By default, Hive will load all SerDe under the hive/lib location. So you are able to do the create/insert/select operations.    In order to read the Hive table created with Custom or external SerDe we need to provide to spark, so spark internally use those libraries and it will load the Hive table data. If you are not provided the serde you can see the following exception: org.apache.hadoop.hive.serde2.SerDeException Please add the following library to the spark-submit command: json-serde-<version>.jar ... View more

Hi @AZIMKBC    Please try to run the SparkPi example and see if is there any error in the logs.    https://rangareddy.github.io/SparkPiExample/   If still issue is not resolved and you are a Cloudera customer please raise a case we will work on internally.    ... View more

Hi @shraddha    Could you please check by any chance if you have set master as local while creating SparkSession in your code.   Use the following sample code to run locally and cluster without updating the master value.  val appName = "MySparkApp" // Creating the SparkConf object val sparkConf = new SparkConf().setAppName(appName).setIfMissing("spark.master", "local[2]") // Creating the SparkSession object val spark: SparkSession = SparkSession.builder().config(sparkConf).getOrCreate()   Verify the whole logs once again to check is there any others errors. ... View more

Hi @Yosieam    Thanks for sharing the code. You forgot to share the spark-submit/pyspark command. Please check what is executor/driver memory is passed to the spark-submit. Could you please confirm file is in local system/hdfs system.   ... View more

Hi @Camilo    When you are sharing the exception you need to share more details. So it will help us to provide a solution in faster way.   1. How are you launching the spark job? 2. If you built application using maven or sbt built tool have you specified spark-hive.jar version. For example, <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hive_2.12</artifactId> <version>2.4.8</version> <scope>provided</scope> </dependency>   References:   1. https://stackoverflow.com/questions/39444493/how-to-create-sparksession-with-hive-support-fails-with-hive-classes-are-not-f 2. https://mvnrepository.com/artifact/org.apache.spark/spark-hive   ... View more

What is the HDP version. if it is HDP3.x then you need to use Hive Warehouse Connector (HWC). ... View more

Hi @mala_etl    You can find the catalog information in the below link:   https://stackoverflow.com/questions/59894454/spark-and-hive-in-hadoop-3-difference-between-metastore-catalog-default-and-spa   Could you please confirm, the table is internal or external table in Hive and also verify the data in Hive. ... View more