I have configured Spark Thrift Server with impersonation. Now the queries run as end user. The "spark.yarn.queue" under "Advanced spark2-thrift-sparkconf" is configured to point to a certain queue ( Say Test_Q).
In my understanding with impersonation, Spark spawns a new container and an STS instance to serve. Now I see for all the users a single queue is used ( Test_Q). We would like that end users have ability to override the queue at run time and not use the same queue for spark thrift server.
If I am correct, we are looking for a property which can override value of
spark.yarn.queue under spark-thrift-sparkconf.conf