https://community.cloudera.com/t5/Support-Questions/Need-Spark-Thrift-Server-Design-because-STS-hang-after/m-p/187014#M149116 <P> <A rel="user" href="https://community.cloudera.com/users/48295/anobido.html" nodeid="48295">@anobi do</A></P><P>For spark driver memory see this link -&gt; <A href="https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-driver.html" target="_blank">https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-driver.html</A></P><P>Also when you do a collect or take, the result comes to driver, your driver will throw error if the result of collect or take is more than free space. Hence it's kept large to account for that if you have big datasets. However default is set to 1G or 2G because it mainly schedules tasks working with YARN with operations being performed on executors themselves (which actually have data, can cache it and process it).</P><P>When you increase sessions, STS daemon memory shall increase too because it has to keep listening and handling sessions.</P><P>My thrift server process was started like this:</P><P><EM><STRONG>hive 27597 13 Nov15 ?00:49:53 /usr/lib/jvm/java-1.8.0/bin/java -Dhdp.version=2.6.1.0-129 -cp /usr/hdp/current/spark2-thriftserver/conf/:/usr/hdp/current/spark2-thriftserver/jars/*:/usr/hdp/current/hadoop-client/conf/ -Xmx6000m org.apache.spark.deploy.SparkSubmit --properties-file /usr/hdp/current/spark2-thriftserver/conf/spark-thrift-sparkconf.conf --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server spark-internal</STRONG></EM></P><P><EM><STRONG><BR /></STRONG></EM></P><P>Note the -Xmx here corresponds to thrift daemon memory rather than driver memory, driver memory is taken from <I>spark2-thriftserver/conf/spark-thrift-sparkconf.conf </I><I> </I>which internally has a symbolic link to one inside /etc.</P><P>If you don't override it there it would just pick default. So please have spark.executor.memory, spark.driver.memory defined there.</P><P>Can you get in your node, do ps -eaf | grep thrift and paste output here?</P><P>I had asked you to set <EM>SPARK_DAEMON_MEMORY=6000m </EM>?</P><P><STRONG>Are you using HDP/Ambari?</STRONG></P><P>If yes, please set it directly here as shown:</P><P><A href="https://community.cloudera.com/legacyfs/online/attachments/43609-screen-shot-2017-11-16-at-104601-am.png">screen-shot-2017-11-16-at-104601-am.png</A></P><P>And set thrift-server parameters here:</P><P><A href="https://community.cloudera.com/legacyfs/online/attachments/43610-screen-shot-2017-11-16-at-104834-am.png">screen-shot-2017-11-16-at-104834-am.png</A></P><P>Just for example.</P><P><STRONG>If you're not using HDP/Ambari, </STRONG></P><P>Set <EM>SPARK_DAEMON_MEMORY </EM>in spark-env.sh and thrift parameters in /etc/spark2/conf/spark-thrift-sparkconf.conf and start thrift-sever.</P><P><EM>spark.driver.cores 1</EM></P><P><EM>spark.driver.memory 40G</EM></P><P><EM>spark.executor.cores 1<BR /></EM></P><P><EM>spark.executor.instances 13<BR /></EM></P><P><EM>spark.executor.memory 40G</EM></P><P>Or you can also give thrift parameters dynamically as mentioned in the IBM link I sent.</P><P><STRONG>You can cross-check your configuration in Environment Tab when you open your application in Spark History Server.</STRONG></P><P>Even I couldn't find a document explaining thrift-server in detail. </P><P>Please confirm that you've done above and cross-check environment in Spark UI.</P> Thu, 16 Nov 2017 13:36:04 GMT tsharma 2017-11-16T13:36:04Z