@anobi do Did you try setting spark.sql.thriftServer.incrementalCollect true?
I am not running multiple queries at a time, so maybe because of that I'm not seeing this, Try decreasing number of simultaneous sessions after setting incremental to true.
Good to hear that anobi. I could not find how to restrict sessions to a particular value.
However if you set this spark.sql.hive.thriftServer.singleSession true.
Only 1 session can be run. This doesn't scale very well.
Please run spark.conf.getAll(), you may find other properties related to num sessions.
Also please accept/upvote any answers if they helped you in concept.
Thank you very much for your support. I changed memory to -Xmx=64g and it seems to resolved my issue. My STS is running about 27 hours for now. I will keep monitoring too see if the problem is resolved permanently. I used to setup -Xmx=25G, then 40G then 56G but STS run for a while and hang. I still do NOT know how to calculate memory needed for STS. I have about 20 user simultaneous.