Hi @Ploeplse    Still, if you are facing the issue, could you share the requested information (i.e code and impala table creation script) ... View more

Hi @Boron    Could you please set the spark-home environment variable like below before creating spark-session. import os os.environ['SPARK_HOME'] = '/usr/hdp/current/spark-client' Reference: https://stackoverflow.com/questions/55569985/pyspark-could-not-find-valid-spark-home https://stackoverflow.com/questions/40087188/cant-find-spark-submit-when-typing-spark-shell ... View more

Hi @Ploeplse  Could you please share reproducible sample code and impala tab creation script? ... View more

Hi @Yosieam    Please avoid calling read_file_log.collect() method. It will bring whole data to the driver and the driver needs to have more memory to hold that much data.   Please check the modified code: move_to_rdd = sc.textFile("datalog2.log").map(lambda row : row.split("time=")).filter(lambda x : x != "") ReSymbol = move_to_rdd.map(lambda x : re.sub(r'\t', ' ', x)).map(lambda x : re.sub(r'\n', ' ', x)).map(lambda x : re.sub(r' +', ' ', x))       ... View more

Hi @mmk    I think you have shared the following information.   7 nodes with each having 250 gb memory and vcpu = 32 per each node   spark-defaults.conf  spark.executor.memory = 100g spark.executor.memoryOverhead = 49g spark.driver.memoryOverhead=200g spark.driver.memory = 500g   You have maximum of 250 gb for node and you have specified driver memory is (500gb and 200gb overhead). How it possible to driver to get 700gb?   Generally you should not exceed the driver/executor memory beyond yarn physical memory.   Coming to the actual problem, please avoid the show() to print 8000000 records. If you need to get the print the all values, then implement a logic to 1000 records at once and next 1000 records for another iteration.    https://stackoverflow.com/questions/29227949/how-to-implement-spark-sql-pagination-query   ... View more

Hi @mmk    By default, Hive will load all SerDe under the hive/lib location. So you are able to do the create/insert/select operations.    In order to read the Hive table created with Custom or external SerDe we need to provide to spark, so spark internally use those libraries and it will load the Hive table data. If you are not provided the serde you can see the following exception: org.apache.hadoop.hive.serde2.SerDeException Please add the following library to the spark-submit command: json-serde-<version>.jar ... View more

Hi @suri789    I think you haven't shared the full code, sample data and expected output to provide a solution. Please share the code proper format.   ... View more

Hi @AZIMKBC    Please try to run the SparkPi example and see if is there any error in the logs.    https://rangareddy.github.io/SparkPiExample/   If still issue is not resolved and you are a Cloudera customer please raise a case we will work on internally.    ... View more