Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Override spark.yarn.queue for Spark Thrift Server

Expert Contributor

Hi,

I have configured Spark Thrift Server with impersonation. Now the queries run as end user. The "spark.yarn.queue" under "Advanced spark2-thrift-sparkconf" is configured to point to a certain queue ( Say Test_Q).

In my understanding with impersonation, Spark spawns a new container and an STS instance to serve. Now I see for all the users a single queue is used ( Test_Q). We would like that end users have ability to override the queue at run time and not use the same queue for spark thrift server.

If I am correct, we are looking for a property which can override value of

spark.yarn.queue under spark-thrift-sparkconf.conf

something like

--queue thequeue for spark-submit

Thanks,

2 REPLIES 2

What is the use case to have user specific YARN queue for STS Jobs? One workaround is to have multiple STS with different queue. How many users are you thinking about?

Expert Contributor

Hi @vshukla, the use case is to let end users use the Queues they generally use to run their interactive queries over other tools like beeline, other jdbc clients.