My cluster is HDP 2.6 with 10 nodes.
I want to run two Spark2 Thrift server with different queue on my cluster. But on Spark2 configure, I just see one option spark.yarn.queue that override both Thrift server configure. That mean both Thrift server are running the same queue.
So anyone have any solution for that I can running two or more Thrift server with different Yarn queue
Hi @Hoang Le , One thing I can think of is , you can try to create two different user group for each of thrift server Ex. Group 1 and Group 2 and then create two queues SparkThrift1 , SparkThrift2 . Assign group1 to SparkThrift1 and group2 to SparkThrift2 queue and see if it is working as you expected.