Support Questions
Find answers, ask questions, and share your expertise

Override spark.yarn.queue for Spark Thrift Server

Expert Contributor


I have configured Spark Thrift Server with impersonation. Now the queries run as end user. The "spark.yarn.queue" under "Advanced spark2-thrift-sparkconf" is configured to point to a certain queue ( Say Test_Q).

In my understanding with impersonation, Spark spawns a new container and an STS instance to serve. Now I see for all the users a single queue is used ( Test_Q). We would like that end users have ability to override the queue at run time and not use the same queue for spark thrift server.

If I am correct, we are looking for a property which can override value of

spark.yarn.queue under spark-thrift-sparkconf.conf

something like

--queue thequeue for spark-submit



What is the use case to have user specific YARN queue for STS Jobs? One workaround is to have multiple STS with different queue. How many users are you thinking about?

Expert Contributor

Hi @vshukla, the use case is to let end users use the Queues they generally use to run their interactive queries over other tools like beeline, other jdbc clients.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.