Support Questions
Find answers, ask questions, and share your expertise

Need Spark Thrift Server Design because STS hang after started about 2 hours

Re: Need Spark Thrift Server Design because STS hang after started about 2 hours

Expert Contributor

@anobi do Did you try setting spark.sql.thriftServer.incrementalCollect true?

I am not running multiple queries at a time, so maybe because of that I'm not seeing this, Try decreasing number of simultaneous sessions after setting incremental to true.

Re: Need Spark Thrift Server Design because STS hang after started about 2 hours

Explorer

Yes @tsharma. I already set it. I have about 20 people use STS to query data everyday. Do you know how to restrict number of simultaneous sessions or restrict maximum memory used per query?

Re: Need Spark Thrift Server Design because STS hang after started about 2 hours

Expert Contributor

Good to hear that anobi. I could not find how to restrict sessions to a particular value.

However if you set this spark.sql.hive.thriftServer.singleSession true.

Only 1 session can be run. This doesn't scale very well.

Please run spark.conf.getAll(), you may find other properties related to num sessions.

Also please accept/upvote any answers if they helped you in concept.

Thank You

Re: Need Spark Thrift Server Design because STS hang after started about 2 hours

Explorer

Thank you very much for your response @tsharma. I do not use Ambari for STS. I will follow your suggestions

Re: Need Spark Thrift Server Design because STS hang after started about 2 hours

Explorer

Hi @tsharma,

Thank you very much for your support. I changed memory to -Xmx=64g and it seems to resolved my issue. My STS is running about 27 hours for now. I will keep monitoring too see if the problem is resolved permanently. I used to setup -Xmx=25G, then 40G then 56G but STS run for a while and hang. I still do NOT know how to calculate memory needed for STS. I have about 20 user simultaneous.