https://community.cloudera.com/t5/Support-Questions/Need-Spark-Thrift-Server-Design-because-STS-hang-after/m-p/187007#M149109 <P>Hi all,</P><P>I am running Spark Thrift Server on Yarn, client mode with 50 executor nodes. First I setup -Xmx=25g for driver, the STS run about 30 mins then hang. After that I increase -Xmx=40G for driver, the STS run about 1 hour then hang. I increase -Xmx=56G for driver, STS run about 2 hours then hang again. I could not keep increasing JVM heap. In all cases, I didn't see any out of memory exception in log file. It seems that when I increased JVM heap on driver STS took most of them. I dumped JVM heap and I saw SparkSession objects are biggest object (one of them is about 10G, others are about 4-6G). I don't understand why SparkSession objects are too large like that. Please:</P><P>1) Is there any suggestion to help me resolve my issue?</P><P>2) Is there any where I can research more about the way STS works. I need a document like <A href="https://cwiki.apache.org/confluence/display/Hive/Design#Design-HiveArchitecture" target="_blank">https://cwiki.apache.org/confluence/display/Hive/Design#Design-HiveArchitecture</A> to understand how STS process query from client because it seems that driver memory keep increasing when there are more people connect and query to STS</P><P>3) How can I sizing memory that need to configure proper for my driver in STS</P><P>Thank you very much,</P> Mon, 13 Nov 2017 22:36:03 GMT anobido 2017-11-13T22:36:03Z