https://community.cloudera.com/t5/Support-Questions/Spark-SQL-Job-stcuk-indefinitely-at-last-task-of-a-stage/m-p/132544#M95214 <P>Hi Puneet --as per suggestion I tried with</P><P>--driver-memory 4g --num-executors 15 --total-executor-cores 30 --executor-memory 10g --driver-cores 2</P>and it failed with <P>Exception in thread "dispatcher-event-loop-3" java.lang.OutOfMemoryError: Java heap space.</P><P>What I am suspecting is parttioning pushing huge data on on one or more executors, and it failes....I saw in spark job environment and <TABLE> <TBODY><TR> <TD>spark.yarn.driver.memoryOverhead = 384</TD> </TR><TR> <TD>spark.yarn.executor.memoryOverhead = 384</TD> </TR></TBODY></TABLE></P>whihc is very low.......i refered documentation and its says <TABLE><TBODY><TR><TD><CODE>spark.yarn.executor.memoryOverhead</CODE></TD><TD>executorMemory * 0.10, with minimum of 384 </TD></TR></TBODY></TABLE> How we can set it to 1G or more Tue, 19 Jul 2016 16:48:30 GMT pkhare 2016-07-19T16:48:30Z