where exectly? ... View more

can i disable leaf queue ? ... View more

no luck, same error, i can only run when setting queue to iliak, instead of root.bt(would be best, cause i want multiple users to post here and enable capacity scheduler with preemption), or root.bt.iliak ... View more

only submitting to queue "iliak" (which located under root.bt.iliak) works, however it does not meed my demands. I want to make a sub queues in bt for each user (for example iliak ) and sub queue is opt root.bt.user1 root.bt.user2 root.bt.user3 root.opt.user1 root.opt.user2 root.opt.user3 because i filed to preemption to work on multiple users on the same queue ... View more

Yes, just made full restart, still no luck The command spark-submit --num-executors 10 --executor-memory 2g --master yarn --deploy-mode cluster --queue root.opt.iliak --conf spark.yarn.submit.waitAppCompletion=false --files run.py The error 18/06/20 12:28:33 INFO Client: Submitting application application_1529419461764_0006 to ResourceManager 18/06/20 12:28:33 INFO Client: Deleted staging directory hdfs://XXXXXXXX:8020/user/iliak/.sparkStaging/application_1529419461764_0006 Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1529419461764_0006 to YARN : Application application_1529419461764_0006 submitted by user iliak to unknown queue: root.opt.iliak at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:272) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:175) at org.apache.spark.deploy.yarn.Client.run(Client.scala:1174) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1233) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:782) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 18/06/20 12:28:34 INFO ShutdownHookManager: Shutdown hook called 18/06/20 12:28:34 INFO ShutdownHookManager: Deleting directory /tmp/spark-a4914501-6859-4fb4-92a3-a6731b62f3a6 ... View more

i added the queue via Yarn queue manager, it not enough ? ... View more

yarn.resourcemanager.scheduler.class = org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler currently Capacity Scheduler values abit differ from what i describe, cause of integration, currently there are 3 queues default 50% bt and opt each at 25% i have done the testing on default queue yarn.scheduler.capacity.maximum-am-resource-percent=0.2 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.root.accessible-node-labels=* yarn.scheduler.capacity.root.acl_administer_queue=yarn yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.default.acl_submit_applications=yarn yarn.scheduler.capacity.root.default.capacity=50 yarn.scheduler.capacity.root.default.maximum-capacity=100 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.user-limit-factor=2 yarn.scheduler.capacity.root.queues=bt,default,opt yarn.scheduler.capacity.queue-mappings-override.enable=false yarn.scheduler.capacity.root.bt.acl_administer_queue=* yarn.scheduler.capacity.root.bt.acl_submit_applications=* yarn.scheduler.capacity.root.bt.capacity=25 yarn.scheduler.capacity.root.bt.maximum-capacity=100 yarn.scheduler.capacity.root.bt.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.bt.ordering-policy=fair yarn.scheduler.capacity.root.bt.ordering-policy.fair.enable-size-based-weight=false yarn.scheduler.capacity.root.bt.priority=0 yarn.scheduler.capacity.root.bt.state=RUNNING yarn.scheduler.capacity.root.bt.user-limit-factor=1 yarn.scheduler.capacity.root.default.acl_administer_queue=yarn yarn.scheduler.capacity.root.default.minimum-user-limit-percent=25 yarn.scheduler.capacity.root.default.ordering-policy=fair yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false yarn.scheduler.capacity.root.default.priority=0 yarn.scheduler.capacity.root.opt.acl_administer_queue=* yarn.scheduler.capacity.root.opt.acl_submit_applications=* yarn.scheduler.capacity.root.opt.capacity=25 yarn.scheduler.capacity.root.opt.maximum-capacity=25 yarn.scheduler.capacity.root.opt.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.opt.ordering-policy=fair yarn.scheduler.capacity.root.opt.ordering-policy.fair.enable-size-based-weight=false yarn.scheduler.capacity.root.opt.priority=0 yarn.scheduler.capacity.root.opt.state=RUNNING yarn.scheduler.capacity.root.opt.user-limit-factor=1 yarn.scheduler.capacity.root.priority=0 ... View more

state accepted ... View more

It is set on fair, but when one user take all the resources and another user submit a job, the second job will not start untill the first job will finish.. for some reason Minimum User Limit have no effect ... View more