Created 03-24-2017 04:18 PM
After configuring below queue's in yarn and after submitting the jobs to created Queue's, the jobs were failing with below error. ERROR: Failed to submit application_XXXXXX to YARN : Application application_XXXXXX submitted by user to unknown queue: root.q01 ## Queue's created in YARN: ---------------------------------------- hdfs@master01:~> mapred queue -list ====================== Queue Name : default Queue State : stopped Scheduling Info : Capacity: 0.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0 ====================== Queue Name : q01 Queue State : running Scheduling Info : Capacity: 50.0, MaximumCapacity: 60.000004, CurrentCapacity: 0.0 ====================== Queue Name : q02 Queue State : running Scheduling Info : Capacity: 50.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0 ====================== Queue Name : child02 Queue State : running Scheduling Info : Capacity: 100.0, MaximumCapacity: 100.0, CurrentCapacity: 0.0 ----------------------------------------- ## Below are the few scenarios, where jobs failed to submit to Q with unknown Queue. Scenario 1: JOB failed to submit, when Q named was provided by appending with parent root Q name(root.q01). -------------------------------------------- set mapred.job.queue.name=root.q01; insert into test_u01 values (1); Failed to submit application_1470318759626_0046 to YARN : Application application_1470318759626_0046 submitted by user user1 to unknown queue: root.q01 Scenario 2: Job executed successfully, when Q named was provided only with child Q(q01). ====================================== set mapred.job.queue.name=q01; insert into test_u01 values (1); INFO : Table default.test_u01 stats: [numFiles=40, numRows=40, totalSize=80, rawDataSize=40] No rows affected (20.125 seconds) ====================================== Scenario 3: Need to execute job into child of a parent Q and where the parent Q is child of root Q. I would to execute the job in child02 which was created in the above Q's but with below Q name it error. ====================================== set mapred.job.queue.name=q02.child02; insert into test_u01 values (1); Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1470318759626_0049 to YARN : Application application_1470318759626_0049 submitted by user user1 to unknown queue: q02.child02 ===================================== Can anyone, please explain how to use Queue names while executing mapreduce jobs and what is the best source to get the actual Q names. And for above scenario 3, what should be the Q name to execute the job successfully in the child Q.
Created 03-24-2017 04:21 PM
Did you refresh the YARN queues?
yarn rmadmin -refreshQueues
Created 03-24-2017 04:27 PM
Am adding the Q's from Yarn Queue Manager, so after adding the Q's am refreshing the capacity scheduler.
Created 03-24-2017 05:12 PM
Can you check the ResourceManager UI that it reflects the newly created queues?
Created 03-24-2017 06:01 PM
I have verified in ResourceManager UI and the queues were in place and i have even pasted the 'mapred queue -list' command output. My concern was to, how the queue names needs to used for running the jobs in to the specified queue but not the queues should to reflected.
Created 03-24-2017 08:20 PM
Didn't scroll thru the code window that had the full context. I believe in Capacity Scheduler (which is what you are using) the queue name is the leaf node name. So instead of using q02.child02 you would simple use child02.
Created 03-27-2017 06:41 PM
Regarding how to specify the YARN queues, this link may be useful: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_yarn_resource_mgt/content/setting_up_que...