Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Run Spark Thrift servers with different Yarn queues

Highlighted

Run Spark Thrift servers with different Yarn queues

My cluster is HDP 2.6 with 10 nodes.

I want to run two Spark2 Thrift server with different queue on my cluster. But on Spark2 configure, I just see one option spark.yarn.queue that override both Thrift server configure. That mean both Thrift server are running the same queue.

So anyone have any solution for that I can running two or more Thrift server with different Yarn queue

Thanks.

3 REPLIES 3
Highlighted

Re: Run Spark Thrift servers with different Yarn queues

@Bob Hardaway

@Sonu Sahi

@Eyad Garelnabi

Do you have any idea for this.

Thanks

Re: Run Spark Thrift servers with different Yarn queues

Explorer

Hi @Hoang Le , One thing I can think of is , you can try to create two different user group for each of thrift server Ex. Group 1 and Group 2 and then create two queues SparkThrift1 , SparkThrift2 . Assign group1 to SparkThrift1 and group2 to SparkThrift2 queue and see if it is working as you expected.

Regards,

Fahim

Highlighted

Re: Run Spark Thrift servers with different Yarn queues

Explorer

@Hoang Le , Please check this thread as well. It may be of help.

https://issues.apache.org/jira/browse/SPARK-18584

Don't have an account?
Coming from Hortonworks? Activate your account here