Hello all, I'm trying to config Hiveserver2 use Spark and it's working perfect with small file. But with large file ( ~ 1.5GB ) , it will be crash by "GC overhead limit exceeded" . My flow is simple like this : 1. Load data from text file into table_text ( text file ~ 1.5G ) Sql: load data local path 'home/abc.txt' into table table_text; 2. select data from table_text to insert to table_orc ( crash in this flow ) SQL : Insert into table table_orc select id,time,data,path,size from table_text; I guess spark have to load all data from table_text and save it in memory before insert to table_orc . I researched and know that spark can config if data does not fit in memory, store the partitions that don't fit on disk, and read them from there when they're needed ( RDD Persistence ). My environment: Ubuntu 16.04 Hive version : 2.3.0 Free memory when launch sql : 4G My config in hive-site.xml: <property> <name>hive.execution.engine</name> <value>spark</value> </property> <property> <name>spark.master</name> <value>local[*]</value> </property> <property> <name>spark.eventLog.enabled</name> <value>true</value> </property> <property> <name>spark.driver.memory</name> <value>12G</value> </property> <property> <name>spark.executor.memory</name> <value>12G</value> </property> <property> <name>spark.serializer</name> <value>org.apache.spark.serializer.KryoSerializer</value> </property> <property> <name>spark.yarn.jars</name> <value>/home/cpu60020-local/Documents/Setup/Java/server/spark/jars/*</value> </property> <property> <name>spark.eventLog.enabled</name> <value>false</value> </property> <property> <name>spark.eventLog.dir</name> <value>/home/cpu60020-local/Documents/Setup/Hive/apache-hive-2.3.0-bin/log/</value> </property> Please tell me if you have any suggess , thanks all ! ... View more