https://community.cloudera.com/t5/Support-Questions/Spark-SQL-Job-stcuk-indefinitely-at-last-task-of-a-stage/m-p/132539#M95209 <P>Thank Puneet for reply..here is my command &amp; other information</P><P>spark-submit --master yarn-client --driver-memory 15g --num-executors 25 --total-executor-cores 60 --executor-memory 15g --driver-cores 2 --conf "spark.executor.memory=-XX:+UseG1GC -XX:+PrintFlagsFinal -XX:+PrintReferenceGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -Xms10g -Xmx10g -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThread=20" --class logicdriver logic.jar</P><P>configuration .....</P><P></P><P>ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"SET hive.execution.engine=tez"</STRONG>); ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"SET hive.optimize.tez=true"</STRONG>); ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"set hive.vectorized.execution.enabled = true "</STRONG>); ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"set hive.vectorized.execution.reduce.enabled = true "</STRONG>); ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"set spark.sql.shuffle.partitions=2050"</STRONG>); ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"SET spark.sql.hive.metastore.version=0.14.0.2.2.4.10-1"</STRONG>); ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"SET hive.warehouse.data.skipTrash=true "</STRONG>); ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"SET hive.exec.dynamic.partition = true "</STRONG>); ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"SET hive.exec.dynamic.partition.mode=nonstrict "</STRONG>); ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"SET spark.driver.maxResultSize= 8192"</STRONG>); ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"SET spark.default.parallelism = 350"</STRONG>); ContextService.<EM>getHiveContext</EM>.sql(<STRONG>"SET spark.yarn.executor.memoryOverhead=1024</STRONG><STRONG>"</STRONG>);</P><P>data---------</P><P>It reads data from from 2 tables and perform join and put result in Dataframes...then again read new tables and does join on previous Dataframe...this cycle goes for 7-8 times and finally it insert result in hive. </P><P>first table has - 63245969 records</P><P>2nd table has - 49275922 records....all the tables have records in this range.</P><P> </P> Mon, 18 Jul 2016 12:37:07 GMT pkhare 2016-07-18T12:37:07Z