Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Capacity Scheduler queue is not working

Capacity Scheduler queue is not working

Explorer

I have added new queue developer and given a capacity 40 %. Below is my change:

yarn.scheduler.capacity.maximum-am-resource-percent=0.2 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.root.accessible-node-labels=* yarn.scheduler.capacity.root.acl_administer_queue=* yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.default.acl_submit_applications=root yarn.scheduler.capacity.root.default.capacity=60 yarn.scheduler.capacity.root.default.maximum-capacity=100 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.user-limit-factor=1 yarn.scheduler.capacity.root.queues=default,developers yarn.scheduler.capacity.queue-mappings-override.enable=false yarn.scheduler.capacity.root.default.priority=0 yarn.scheduler.capacity.root.developers.acl_administer_queue=* yarn.scheduler.capacity.root.developers.acl_submit_applications=hdfs yarn.scheduler.capacity.root.developers.capacity=40 yarn.scheduler.capacity.root.developers.maximum-capacity=40 yarn.scheduler.capacity.root.developers.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.developers.ordering-policy=fifo yarn.scheduler.capacity.root.developers.priority=0 yarn.scheduler.capacity.root.developers.state=RUNNING yarn.scheduler.capacity.root.developers.user-limit-factor=1 yarn.scheduler.capacity.root.priority=0

When i execute query from hdfs user that belong to developer group then it is going in default queue instead of developer.

Please note that i am using hdp 2.5 and ambari 2.6 version,

2 REPLIES 2

Re: Capacity Scheduler queue is not working

Expert Contributor

@Fahad Sarwar

The "acl_submit_applications" property controls who can (has permission to) submit applications to the given queue but it in itself doesn't say anything about which queue to actually use.

Defining which YARN queue to use is application specific, for example in MapReduce, you need this property: mapreduce.job.queuename.

Re: Capacity Scheduler queue is not working

Guru

@Fahad Sarwar, Capacity scheduler does not have placement rules, where you can configure that if userX is running job, place the job in QueueX.

Capacity scheduler ACLS only check whether the application is allowed to run in specific queue or not.

In order to be able to run the job in a specific queue, you will need to set the queue config while running apps. ( If the queue config is not set, by default application gets launched in "default" queue)

For mapreduce jobs : set -Dmapred.job.queue.name=<queue-name> or -Dmapred.job.queuename=<queue-name>

yarn jar /usr/lib/gphd/hadoop-mapreduce/hadoop-mapreduce-examples-x.x.x-alpha-gphd-x.x.x.x.jar wordcount -D mapreduce.job.queuename=<queue-name> /tmp/test_input /user/fail_user/test_output

For spark jobs: set --queue <queue-name>

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --queue <queue-name> /usr/hdp/2.x.x.x-xxxx/spark/lib/spark-examples-x.x.x.x.x.x.x-xxxx-hadoopx.x.x.x.x.x.x-xxxx.jar 10
Don't have an account?
Coming from Hortonworks? Activate your account here