spark submit Exception unknown queue

[#1]

spark-submit --num-executors 10 --executor-memory 2g --master yarn --deploy-mode cluster --queue root.opt.iliak --conf spark.yarn.submit.waitAppCompletion=false --files run.py 

gives error

Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1529419461764_0004 to YARN : Application application_1529419461764_0004 submitted by user iliak to unknown queue: root.opt.iliak

while submiting to queue "default" works (root.defult queue exists)

[#2]

• root.bt.user1 (here the leaf queue is user1) , yes we can disable the leaf queue. Option will be available in ranger (Since you are using ranger)

[#3]

where exectly?

[#4]

I think this is completely off from what it is described in the question description. If disabling queue is your current goal.

Goto Ranger admin UI -> Service Manager -> Click Yarn -> Create/edit existing policy. There you will find toggle to enable/disable, Choose disable.

[#5]

I found a working dirty solution, added the queues as follow:

• root.bt.bt_user1
• root.bt.bt_user2
• root.bt.bt_user3
• root.opt.opt_user1
• root.opt.opt_user2
• root.opt.opt_user3

and updated the submission script to match it