spark submit Exception unknown queue

[#1]

spark-submit --num-executors 10 --executor-memory 2g --master yarn --deploy-mode cluster --queue root.opt.iliak --conf spark.yarn.submit.waitAppCompletion=false --files run.py 

gives error

Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1529419461764_0004 to YARN : Application application_1529419461764_0004 submitted by user iliak to unknown queue: root.opt.iliak

while submiting to queue "default" works (root.defult queue exists)

[#2]

@IIia K

The Error states that you don't have the queue named "root.opt.iliak", Do check in yarn if the queue really exists.

[#3]

i added the queue via Yarn queue manager, it not enough ?

[#4]

It should be enough. Are you still seeing the same issue?

[#5]

Yes, just made full restart, still no luck

The command

spark-submit --num-executors 10 --executor-memory 2g --master yarn --deploy-mode cluster --queue root.opt.iliak --conf spark.yarn.submit.waitAppCompletion=false --files run.py

The error

18/06/20 12:28:33 INFO Client: Submitting application application_1529419461764_0006 to ResourceManager
18/06/20 12:28:33 INFO Client: Deleted staging directory hdfs://XXXXXXXX:8020/user/iliak/.sparkStaging/application_1529419461764_0006
Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1529419461764_0006 to YARN : Application application_1529419461764_0006 submitted by user iliak to unknown queue: root.opt.iliak
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:272)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:175)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1174)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1233)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:782)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/06/20 12:28:34 INFO ShutdownHookManager: Shutdown hook called
18/06/20 12:28:34 INFO ShutdownHookManager: Deleting directory /tmp/spark-a4914501-6859-4fb4-92a3-a6731b62f3a6

[#6]

only submitting to queue "iliak" (which located under root.bt.iliak) works, however it does not meed my demands.

I want to make a sub queues in bt for each user (for example iliak ) and sub queue is opt

• root.bt.user1
• root.bt.user2
• root.bt.user3
• root.opt.user1
• root.opt.user2
• root.opt.user3

because i filed to preemption to work on multiple users on the same queue

[#7]

@Ilia K Try switching to user yarn and running

yarn rmadmin -refreshQueues

See if the above command executes successfully. Then run as follows (note I use iliak for queue name only):

spark-submit --num-executors 10--executor-memory 2g--master yarn --deploy-mode cluster --queue iliak --conf spark.yarn.submit.waitAppCompletion=false--files run.py 

Another thing you could try is to switch the ordering policy to Empty, save and test again.

HTH

[#8]

no luck, same error, i can only run when setting queue to iliak, instead of root.bt(would be best, cause i want multiple users to post here and enable capacity scheduler with preemption), or root.bt.iliak

[#9]

@Ilia K

I guess using "--queue iliak" is the only way to run the app in that leaf queue. Also note that we cannot create multiple hierarchy with same queue name like below. user1 can be defined only once.

• root.bt.user1
• root.bt.user2
• root.bt.user3
• root.opt.user1
• root.opt.user2
• root.opt.user3

[#10]

can i disable leaf queue ?

[#11]

• root.bt.user1 (here the leaf queue is user1) , yes we can disable the leaf queue. Option will be available in ranger (Since you are using ranger)

[#12]

where exectly?

[#13]

I think this is completely off from what it is described in the question description. If disabling queue is your current goal.

Goto Ranger admin UI -> Service Manager -> Click Yarn -> Create/edit existing policy. There you will find toggle to enable/disable, Choose disable.

[#14]

I found a working dirty solution, added the queues as follow:

• root.bt.bt_user1
• root.bt.bt_user2
• root.bt.bt_user3
• root.opt.opt_user1
• root.opt.opt_user2
• root.opt.opt_user3

and updated the submission script to match it