Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

How yarn queue names should be used while executing the mapreduce jobs.

Explorer
After configuring below queue's in yarn and after submitting the jobs to created Queue's, the jobs were failing with below error.
ERROR:
Failed to submit application_XXXXXX to YARN : Application application_XXXXXX submitted by user  to unknown queue: root.q01
## Queue's created in YARN:
----------------------------------------
hdfs@master01:~> mapred queue -list
======================
Queue Name : default
Queue State : stopped
Scheduling Info : Capacity: 0.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0
======================
Queue Name : q01
Queue State : running
Scheduling Info : Capacity: 50.0, MaximumCapacity: 60.000004, CurrentCapacity: 0.0
======================
Queue Name : q02
Queue State : running
Scheduling Info : Capacity: 50.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0
    ======================
    Queue Name : child02
    Queue State : running
    Scheduling Info : Capacity: 100.0, MaximumCapacity: 100.0, CurrentCapacity: 0.0
-----------------------------------------
## Below are the few scenarios, where jobs failed to submit to Q with unknown Queue.
Scenario 1: JOB failed to submit, when Q named was provided by appending with parent root Q name(root.q01).
--------------------------------------------
set mapred.job.queue.name=root.q01;
insert into test_u01 values (1);
Failed to submit application_1470318759626_0046 to YARN : Application application_1470318759626_0046 submitted by user user1 to unknown queue: root.q01
Scenario 2: Job executed successfully, when Q named was provided only with child Q(q01).
======================================
set mapred.job.queue.name=q01;
insert into test_u01 values (1);
INFO  : Table default.test_u01 stats: [numFiles=40, numRows=40, totalSize=80, rawDataSize=40]
No rows affected (20.125 seconds)
======================================
Scenario 3: Need to execute job into child of a parent Q and where the parent Q is child of root Q. I would to execute the job in child02 which was created in the above Q's but with below Q name it error.
======================================
set mapred.job.queue.name=q02.child02;
insert into test_u01 values (1);
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1470318759626_0049 to YARN : Application application_1470318759626_0049 submitted by user user1 to unknown queue: q02.child02
=====================================
Can anyone, please explain how to use Queue names while executing mapreduce jobs and what is the best source to get the actual Q names. And for above scenario 3, what should be the Q name to execute the job successfully in the child Q.
6 REPLIES 6

Master Collaborator

Did you refresh the YARN queues?

yarn rmadmin -refreshQueues

Explorer
@Deepesh

Am adding the Q's from Yarn Queue Manager, so after adding the Q's am refreshing the capacity scheduler.

Master Collaborator

Can you check the ResourceManager UI that it reflects the newly created queues?

Explorer

@Deepesh

I have verified in ResourceManager UI and the queues were in place and i have even pasted the 'mapred queue -list' command output. My concern was to, how the queue names needs to used for running the jobs in to the specified queue but not the queues should to reflected.

Master Collaborator

Didn't scroll thru the code window that had the full context. I believe in Capacity Scheduler (which is what you are using) the queue name is the leaf node name. So instead of using q02.child02 you would simple use child02.

Explorer

Regarding how to specify the YARN queues, this link may be useful: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_yarn_resource_mgt/content/setting_up_que...