Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

spark submit Exception unknown queue

avatar
Rising Star
spark-submit --num-executors 10 --executor-memory 2g --master yarn --deploy-mode cluster --queue root.opt.iliak --conf spark.yarn.submit.waitAppCompletion=false --files run.py 

gives error

Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1529419461764_0004 to YARN : Application application_1529419461764_0004 submitted by user iliak to unknown queue: root.opt.iliak

while submiting to queue "default" works (root.defult queue exists)

13 REPLIES 13

avatar
@IIia K

The Error states that you don't have the queue named "root.opt.iliak", Do check in yarn if the queue really exists.

avatar
Rising Star

i added the queue via Yarn queue manager, it not enough ?

avatar

It should be enough. Are you still seeing the same issue?

avatar
Rising Star

Yes, just made full restart, still no luck

The command

spark-submit --num-executors 10 --executor-memory 2g --master yarn --deploy-mode cluster --queue root.opt.iliak --conf spark.yarn.submit.waitAppCompletion=false --files run.py

The error

18/06/20 12:28:33 INFO Client: Submitting application application_1529419461764_0006 to ResourceManager
18/06/20 12:28:33 INFO Client: Deleted staging directory hdfs://XXXXXXXX:8020/user/iliak/.sparkStaging/application_1529419461764_0006
Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1529419461764_0006 to YARN : Application application_1529419461764_0006 submitted by user iliak to unknown queue: root.opt.iliak
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:272)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:175)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1174)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1233)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:782)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/06/20 12:28:34 INFO ShutdownHookManager: Shutdown hook called
18/06/20 12:28:34 INFO ShutdownHookManager: Deleting directory /tmp/spark-a4914501-6859-4fb4-92a3-a6731b62f3a6

77778-ambari-queue.png

avatar
Rising Star

only submitting to queue "iliak" (which located under root.bt.iliak) works, however it does not meed my demands.

I want to make a sub queues in bt for each user (for example iliak ) and sub queue is opt

  • root.bt.user1
  • root.bt.user2
  • root.bt.user3
  • root.opt.user1
  • root.opt.user2
  • root.opt.user3

because i filed to preemption to work on multiple users on the same queue

avatar

@Ilia K Try switching to user yarn and running

yarn rmadmin -refreshQueues

See if the above command executes successfully. Then run as follows (note I use iliak for queue name only):

spark-submit --num-executors 10--executor-memory 2g--master yarn --deploy-mode cluster --queue iliak --conf spark.yarn.submit.waitAppCompletion=false--files run.py 

Another thing you could try is to switch the ordering policy to Empty, save and test again.

HTH

avatar
Rising Star

no luck, same error, i can only run when setting queue to iliak, instead of root.bt(would be best, cause i want multiple users to post here and enable capacity scheduler with preemption), or root.bt.iliak

avatar
@Ilia K

I guess using "--queue iliak" is the only way to run the app in that leaf queue. Also note that we cannot create multiple hierarchy with same queue name like below. user1 can be defined only once.

  • root.bt.user1
  • root.bt.user2
  • root.bt.user3
  • root.opt.user1
  • root.opt.user2
  • root.opt.user3

avatar
Rising Star

can i disable leaf queue ?