Override spark.yarn.queue for Spark Thrift Server

[#1]

Hi,

I have configured Spark Thrift Server with impersonation. Now the queries run as end user. The "spark.yarn.queue" under "Advanced spark2-thrift-sparkconf" is configured to point to a certain queue ( Say Test_Q).

In my understanding with impersonation, Spark spawns a new container and an STS instance to serve. Now I see for all the users a single queue is used ( Test_Q). We would like that end users have ability to override the queue at run time and not use the same queue for spark thrift server.

If I am correct, we are looking for a property which can override value of

spark.yarn.queue under spark-thrift-sparkconf.conf

something like

--queue thequeue for spark-submit

Thanks,

[#2]

What is the use case to have user specific YARN queue for STS Jobs? One workaround is to have multiple STS with different queue. How many users are you thinking about?

[#3]

Hi @vshukla, the use case is to let end users use the Queues they generally use to run their interactive queries over other tools like beeline, other jdbc clients.