Support Questions
Find answers, ask questions, and share your expertise

How multiple users run spark-sql query concurrently?

Hi, All

I run spark-sql queries with a thrift server.

I know that if multiple sql queries are submitted through the thrift server each query would be run sequentially.

If many users want to query the table on a spark cluster over yarn at the same time, how these requested queries could be run concurrently?

The requested query do not update the table and just query

I have an idea that because a thrift server has dedicated executor cluster if multiple thrift servers are used multiple queries could be processed concurrently.

Is there any idea about this situation?

Thanks in advance.

Park.

1 ACCEPTED SOLUTION

Accepted Solutions

@Young-Geun Park

Have you taken a look at http://spark.apache.org/docs/1.6.2/job-scheduling.html? Also, if you start the thrift server in yarn-client mode, you should be able to take advantage of YARN resource scheduling and queues.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/startin...

View solution in original post

1 REPLY 1

@Young-Geun Park

Have you taken a look at http://spark.apache.org/docs/1.6.2/job-scheduling.html? Also, if you start the thrift server in yarn-client mode, you should be able to take advantage of YARN resource scheduling and queues.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/startin...

View solution in original post