https://community.cloudera.com/t5/Support-Questions/Spark-SQL-Job-stcuk-indefinitely-at-last-task-of-a-stage/m-p/132540#M95210 <P>Hi Praveen,</P><P>Here are a few points to help:</P><P>1. Try running your API without options like "--driver-memory 15g --num-executors 25 --total-executor-cores 60 --executor-memory 15g --driver-cores 2" and check logs for memory allocated to RDDs/DataFrames.</P><P>2. Driver doesn't need 15g memory if you are not collecting data on driver. Try setting it to 4g rather. I hope u r not using .collect() or similar operations which collect all data to driver.</P><P>3. The error needs fine tuning your configurations between executor memory and driver memory. The total number of executors(25) are pretty much higher considering the memory allocated(15g). Reduce number of executors and consider allocating less memory(4g to start with).</P><P>Thanks,</P><P>Puneet</P> Mon, 18 Jul 2016 15:09:49 GMT psingh15 2016-07-18T15:09:49Z