question Re: All spark-submit are routed to the same yarn queue in Support Questions

All spark-submit are routed to the same yarn queue

Samie — Thu, 15 Dec 2022 12:20:17 GMT

Hi,

we are using HDP 2.6.5 and are facing an issue for some days now.
Yesterday we realized that when we issue a spark-submit command, it is sent to the same yarn queue even if we specify --queue param.

Had anybody ever come to that issue?

Spark version is 2.3.0.

Regards

Re: All spark-submit are routed to the same yarn queue

RangaReddy — Fri, 16 Dec 2022 05:13:37 GMT

HI @Samie

Please attach the spark application and event logs to check the queue name. The easiest way to check the spark application is by running spark pi example.

spark-submit \ --class org.apache.spark.examples.SparkPi \ --queue <queue_name> \ --master yarn \ --deploy-mode cluster \ --num-executors 1 \ --driver-memory 512m \ --executor-memory 512m \ --executor-cores 1 \ /usr/hdp/current/spark2-client/examples/jars/spark-examples_*.jar 10

Spark on YARN only: --queue QUEUE_NAME The YARN queue to submit to (Default: "default").

Re: All spark-submit are routed to the same yarn queue

RangaReddy — Wed, 21 Dec 2022 06:18:39 GMT

Hi @Samie

Is there any update on your testing?

Re: All spark-submit are routed to the same yarn queue

VidyaSargur — Mon, 02 Jan 2023 08:05:59 GMT

@Samie, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.