Run Spark Thrift servers with different Yarn queues

[#1]

My cluster is HDP 2.6 with 10 nodes.

I want to run two Spark2 Thrift server with different queue on my cluster. But on Spark2 configure, I just see one option spark.yarn.queue that override both Thrift server configure. That mean both Thrift server are running the same queue.

So anyone have any solution for that I can running two or more Thrift server with different Yarn queue

Thanks.

[#2]

@Bob Hardaway

@Sonu Sahi

@Eyad Garelnabi

Do you have any idea for this.

Thanks

[#3]

Hi @Hoang Le , One thing I can think of is , you can try to create two different user group for each of thrift server Ex. Group 1 and Group 2 and then create two queues SparkThrift1 , SparkThrift2 . Assign group1 to SparkThrift1 and group2 to SparkThrift2 queue and see if it is working as you expected.

Regards,

Fahim

[#4]

@Hoang Le , Please check this thread as well. It may be of help.

https://issues.apache.org/jira/browse/SPARK-18584