- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How yarn queue names should be used while executing the mapreduce jobs.
- Labels:
-
Apache Hive
-
Apache YARN
Created ‎03-24-2017 04:18 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
After configuring below queue's in yarn and after submitting the jobs to created Queue's, the jobs were failing with below error. ERROR: Failed to submit application_XXXXXX to YARN : Application application_XXXXXX submitted by user to unknown queue: root.q01 ## Queue's created in YARN: ---------------------------------------- hdfs@master01:~> mapred queue -list ====================== Queue Name : default Queue State : stopped Scheduling Info : Capacity: 0.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0 ====================== Queue Name : q01 Queue State : running Scheduling Info : Capacity: 50.0, MaximumCapacity: 60.000004, CurrentCapacity: 0.0 ====================== Queue Name : q02 Queue State : running Scheduling Info : Capacity: 50.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0 ====================== Queue Name : child02 Queue State : running Scheduling Info : Capacity: 100.0, MaximumCapacity: 100.0, CurrentCapacity: 0.0 ----------------------------------------- ## Below are the few scenarios, where jobs failed to submit to Q with unknown Queue. Scenario 1: JOB failed to submit, when Q named was provided by appending with parent root Q name(root.q01). -------------------------------------------- set mapred.job.queue.name=root.q01; insert into test_u01 values (1); Failed to submit application_1470318759626_0046 to YARN : Application application_1470318759626_0046 submitted by user user1 to unknown queue: root.q01 Scenario 2: Job executed successfully, when Q named was provided only with child Q(q01). ====================================== set mapred.job.queue.name=q01; insert into test_u01 values (1); INFO : Table default.test_u01 stats: [numFiles=40, numRows=40, totalSize=80, rawDataSize=40] No rows affected (20.125 seconds) ====================================== Scenario 3: Need to execute job into child of a parent Q and where the parent Q is child of root Q. I would to execute the job in child02 which was created in the above Q's but with below Q name it error. ====================================== set mapred.job.queue.name=q02.child02; insert into test_u01 values (1); Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1470318759626_0049 to YARN : Application application_1470318759626_0049 submitted by user user1 to unknown queue: q02.child02 ===================================== Can anyone, please explain how to use Queue names while executing mapreduce jobs and what is the best source to get the actual Q names. And for above scenario 3, what should be the Q name to execute the job successfully in the child Q.
Created ‎03-24-2017 04:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Did you refresh the YARN queues?
yarn rmadmin -refreshQueues
Created ‎03-24-2017 04:27 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Am adding the Q's from Yarn Queue Manager, so after adding the Q's am refreshing the capacity scheduler.
Created ‎03-24-2017 05:12 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you check the ResourceManager UI that it reflects the newly created queues?
Created ‎03-24-2017 06:01 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have verified in ResourceManager UI and the queues were in place and i have even pasted the 'mapred queue -list' command output. My concern was to, how the queue names needs to used for running the jobs in to the specified queue but not the queues should to reflected.
Created ‎03-24-2017 08:20 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Didn't scroll thru the code window that had the full context. I believe in Capacity Scheduler (which is what you are using) the queue name is the leaf node name. So instead of using q02.child02 you would simple use child02.
Created ‎03-27-2017 06:41 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Regarding how to specify the YARN queues, this link may be useful: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_yarn_resource_mgt/content/setting_up_que...
