https://community.cloudera.com/t5/Support-Questions/Need-Spark-Thrift-Server-Design-because-STS-hang-after/m-p/187015#M149117 <P>You may also be facing a bug. Check below links and your spark version.</P><P><A href="https://issues.apache.org/jira/browse/SPARK-18857">https://issues.apache.org/jira/browse/SPARK-18857</A></P><P><A href="https://forums.databricks.com/questions/344/how-does-the-jdbc-odbc-thrift-server-stream-query.html">https://forums.databricks.com/questions/344/how-does-the-jdbc-odbc-thrift-server-stream-query.html</A></P><P><A href="https://stackoverflow.com/questions/35046692/spark-incremental-collect-to-a-partition-causes-outofmemory-in-heap">https://stackoverflow.com/questions/35046692/spark-incremental-collect-to-a-partition-causes-outofmemory-in-heap</A></P><P>Regardless, please try with <EM>spark.sql.thriftServer.incrementalCollect true </EM>in thrift conf or start thrift-server with that.</P><P>It is set to False by default, this would be an important thing to check and has a direct implication on driver heap (if you're in fact running out of that). Read link below:</P><P><A href="http://www.russellspitzer.com/2017/05/19/Spark-Sql-Thriftserver/" target="_blank">http://www.russellspitzer.com/2017/05/19/Spark-Sql-Thriftserver/</A></P> Thu, 16 Nov 2017 13:59:06 GMT tsharma 2017-11-16T13:59:06Z