Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How yarn queue names should be used while executing the mapreduce jobs.

avatar
Contributor
After configuring below queue's in yarn and after submitting the jobs to created Queue's, the jobs were failing with below error.
ERROR:
Failed to submit application_XXXXXX to YARN : Application application_XXXXXX submitted by user  to unknown queue: root.q01
## Queue's created in YARN:
----------------------------------------
hdfs@master01:~> mapred queue -list
======================
Queue Name : default
Queue State : stopped
Scheduling Info : Capacity: 0.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0
======================
Queue Name : q01
Queue State : running
Scheduling Info : Capacity: 50.0, MaximumCapacity: 60.000004, CurrentCapacity: 0.0
======================
Queue Name : q02
Queue State : running
Scheduling Info : Capacity: 50.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0
    ======================
    Queue Name : child02
    Queue State : running
    Scheduling Info : Capacity: 100.0, MaximumCapacity: 100.0, CurrentCapacity: 0.0
-----------------------------------------
## Below are the few scenarios, where jobs failed to submit to Q with unknown Queue.
Scenario 1: JOB failed to submit, when Q named was provided by appending with parent root Q name(root.q01).
--------------------------------------------
set mapred.job.queue.name=root.q01;
insert into test_u01 values (1);
Failed to submit application_1470318759626_0046 to YARN : Application application_1470318759626_0046 submitted by user user1 to unknown queue: root.q01
Scenario 2: Job executed successfully, when Q named was provided only with child Q(q01).
======================================
set mapred.job.queue.name=q01;
insert into test_u01 values (1);
INFO  : Table default.test_u01 stats: [numFiles=40, numRows=40, totalSize=80, rawDataSize=40]
No rows affected (20.125 seconds)
======================================
Scenario 3: Need to execute job into child of a parent Q and where the parent Q is child of root Q. I would to execute the job in child02 which was created in the above Q's but with below Q name it error.
======================================
set mapred.job.queue.name=q02.child02;
insert into test_u01 values (1);
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1470318759626_0049 to YARN : Application application_1470318759626_0049 submitted by user user1 to unknown queue: q02.child02
=====================================
Can anyone, please explain how to use Queue names while executing mapreduce jobs and what is the best source to get the actual Q names. And for above scenario 3, what should be the Q name to execute the job successfully in the child Q.
6 REPLIES 6

avatar

Did you refresh the YARN queues?

yarn rmadmin -refreshQueues

avatar
Contributor
@Deepesh

Am adding the Q's from Yarn Queue Manager, so after adding the Q's am refreshing the capacity scheduler.

avatar

Can you check the ResourceManager UI that it reflects the newly created queues?

avatar
Contributor

@Deepesh

I have verified in ResourceManager UI and the queues were in place and i have even pasted the 'mapred queue -list' command output. My concern was to, how the queue names needs to used for running the jobs in to the specified queue but not the queues should to reflected.

avatar

Didn't scroll thru the code window that had the full context. I believe in Capacity Scheduler (which is what you are using) the queue name is the leaf node name. So instead of using q02.child02 you would simple use child02.

avatar
Explorer

Regarding how to specify the YARN queues, this link may be useful: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_yarn_resource_mgt/content/setting_up_que...