In hive, when doas set to true, hive jobs are running as enduser(user who is executing the job) and yarn acl's set for queue are in affect but when doas set to false, all the hive jobs were run as hive user and yarn acl's are not in affect on enduser running the job. In below scenario(doas set to false), user 'user02' when running job in 'engineering01' Queue, where only 'user02' can submit application in Q, but hive job is failing with "User hive cannot submit applications to queue root.engineering01" error. In this scenario, how does yarn acl's will affect for enduser and granting hive user to submit application is every queue is not applicable. ================ master01:~ # su - user02 ----------------- user02@master01:/root> mapred queue -showacls 17/10/15 19:42:27 INFO impl.TimelineClientImpl: Timeline service address: http://master01.teradata.com:8188/ws/v1/timeline/ 17/10/15 19:42:28 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 Queue acls for user : user02 Queue Operations ===================== root default SUBMIT_APPLICATIONS engineering01 SUBMIT_APPLICATIONS support01 ADMINISTER_QUEUE,SUBMIT_APPLICATIONS -------------------- user02@master01:/root> beeline -u "jdbc:hive2://localhost:10000/default" -n user02 -p user02 WARNING: Use "yarn jar" to launch YARN applications. Connecting to jdbc:hive2://localhost:10000/default Connected to: Apache Hive (version 1.2.1.2.3.4.0-3485) Driver: Hive JDBC (version 1.2.1.2.3.4.0-3485) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.2.1.2.3.4.0-3485 by Apache Hive [INFO] Unable to bind key for unsupported operation: backward-delete-word [INFO] Unable to bind key for unsupported operation: backward-delete-word [INFO] Unable to bind key for unsupported operation: down-history [INFO] Unable to bind key for unsupported operation: up-history [INFO] Unable to bind key for unsupported operation: up-history [INFO] Unable to bind key for unsupported operation: down-history [INFO] Unable to bind key for unsupported operation: up-history [INFO] Unable to bind key for unsupported operation: down-history [INFO] Unable to bind key for unsupported operation: up-history [INFO] Unable to bind key for unsupported operation: down-history [INFO] Unable to bind key for unsupported operation: up-history [INFO] Unable to bind key for unsupported operation: down-history 0: jdbc:hive2://localhost:10000/default> set tez.queue.name=engineering01; No rows affected (0.061 seconds) 0: jdbc:hive2://localhost:10000/default> create table test09 as select * from employee01; INFO : Tez session hasn't been created yet. Opening session ERROR : Failed to execute tez graph. org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1508070646645_0021 to YARN : org.apache.hadoop.security.AccessControlException: User hive cannot submit applications to queue root.engineering01 at org.apache.tez.client.TezClient.start(TezClient.java:413) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196) at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1703) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1460) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1096) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154) at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71) at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1508070646645_0021 to YARN : org.apache.hadoop.security.AccessControlException: User hive cannot submit applications to queue root.engineering01 at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271) at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72) at org.apache.tez.client.TezClient.start(TezClient.java:408) ... 22 more Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1) 0: jdbc:hive2://localhost:10000/default> ... View more

After configuring below queue's in yarn and after submitting the jobs to created Queue's, the jobs were failing with below error. ERROR: Failed to submit application_XXXXXX to YARN : Application application_XXXXXX submitted by user to unknown queue: root.q01 ## Queue's created in YARN: ---------------------------------------- hdfs@master01:~> mapred queue -list ====================== Queue Name : default Queue State : stopped Scheduling Info : Capacity: 0.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0 ====================== Queue Name : q01 Queue State : running Scheduling Info : Capacity: 50.0, MaximumCapacity: 60.000004, CurrentCapacity: 0.0 ====================== Queue Name : q02 Queue State : running Scheduling Info : Capacity: 50.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0 ====================== Queue Name : child02 Queue State : running Scheduling Info : Capacity: 100.0, MaximumCapacity: 100.0, CurrentCapacity: 0.0 ----------------------------------------- ## Below are the few scenarios, where jobs failed to submit to Q with unknown Queue. Scenario 1: JOB failed to submit, when Q named was provided by appending with parent root Q name(root.q01). -------------------------------------------- set mapred.job.queue.name=root.q01; insert into test_u01 values (1); Failed to submit application_1470318759626_0046 to YARN : Application application_1470318759626_0046 submitted by user user1 to unknown queue: root.q01 Scenario 2: Job executed successfully, when Q named was provided only with child Q(q01). ====================================== set mapred.job.queue.name=q01; insert into test_u01 values (1); INFO : Table default.test_u01 stats: [numFiles=40, numRows=40, totalSize=80, rawDataSize=40] No rows affected (20.125 seconds) ====================================== Scenario 3: Need to execute job into child of a parent Q and where the parent Q is child of root Q. I would to execute the job in child02 which was created in the above Q's but with below Q name it error. ====================================== set mapred.job.queue.name=q02.child02; insert into test_u01 values (1); Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1470318759626_0049 to YARN : Application application_1470318759626_0049 submitted by user user1 to unknown queue: q02.child02 ===================================== Can anyone, please explain how to use Queue names while executing mapreduce jobs and what is the best source to get the actual Q names. And for above scenario 3, what should be the Q name to execute the job successfully in the child Q. ... View more