We have Ranger installed on hadoop cluster and had to set doas to false, due to which all the hive jobs running from end users where shown as hive user in RM webui and in Ranger audit. Due to which, we are unable to do auditing. We would like to set doas to true, are their any impacts on cluster or on any other services/components for setting doas to true. ... View more

While i was installing HDP 2.5.3, in the 'install services' section, i was asked to select below services on nodes in the cluster. But am not sure about below services to be installed on what node(Master, data, edge). 1. Phoenix Query Server 2. Supervisor 3. Flume 4. Accumulo TServer 5. Livy Server 6. Spark Thrift Server Could any one clarify, the services to be installed on what type of node and on how nodes does the service needs to be running. ... View more

In hive, when doas set to true, hive jobs are running as enduser(user who is executing the job) and yarn acl's set for queue are in affect but when doas set to false, all the hive jobs were run as hive user and yarn acl's are not in affect on enduser running the job. In below scenario(doas set to false), user 'user02' when running job in 'engineering01' Queue, where only 'user02' can submit application in Q, but hive job is failing with "User hive cannot submit applications to queue root.engineering01" error. In this scenario, how does yarn acl's will affect for enduser and granting hive user to submit application is every queue is not applicable. ================ master01:~ # su - user02 ----------------- user02@master01:/root> mapred queue -showacls 17/10/15 19:42:27 INFO impl.TimelineClientImpl: Timeline service address: http://master01.teradata.com:8188/ws/v1/timeline/ 17/10/15 19:42:28 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 Queue acls for user : user02 Queue Operations ===================== root default SUBMIT_APPLICATIONS engineering01 SUBMIT_APPLICATIONS support01 ADMINISTER_QUEUE,SUBMIT_APPLICATIONS -------------------- user02@master01:/root> beeline -u "jdbc:hive2://localhost:10000/default" -n user02 -p user02 WARNING: Use "yarn jar" to launch YARN applications. Connecting to jdbc:hive2://localhost:10000/default Connected to: Apache Hive (version 1.2.1.2.3.4.0-3485) Driver: Hive JDBC (version 1.2.1.2.3.4.0-3485) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.2.1.2.3.4.0-3485 by Apache Hive [INFO] Unable to bind key for unsupported operation: backward-delete-word [INFO] Unable to bind key for unsupported operation: backward-delete-word [INFO] Unable to bind key for unsupported operation: down-history [INFO] Unable to bind key for unsupported operation: up-history [INFO] Unable to bind key for unsupported operation: up-history [INFO] Unable to bind key for unsupported operation: down-history [INFO] Unable to bind key for unsupported operation: up-history [INFO] Unable to bind key for unsupported operation: down-history [INFO] Unable to bind key for unsupported operation: up-history [INFO] Unable to bind key for unsupported operation: down-history [INFO] Unable to bind key for unsupported operation: up-history [INFO] Unable to bind key for unsupported operation: down-history 0: jdbc:hive2://localhost:10000/default> set tez.queue.name=engineering01; No rows affected (0.061 seconds) 0: jdbc:hive2://localhost:10000/default> create table test09 as select * from employee01; INFO : Tez session hasn't been created yet. Opening session ERROR : Failed to execute tez graph. org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1508070646645_0021 to YARN : org.apache.hadoop.security.AccessControlException: User hive cannot submit applications to queue root.engineering01 at org.apache.tez.client.TezClient.start(TezClient.java:413) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196) at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1703) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1460) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1096) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154) at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71) at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1508070646645_0021 to YARN : org.apache.hadoop.security.AccessControlException: User hive cannot submit applications to queue root.engineering01 at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271) at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72) at org.apache.tez.client.TezClient.start(TezClient.java:408) ... 22 more Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1) 0: jdbc:hive2://localhost:10000/default> ... View more

When i try to execute below hive rest api to access hive tables, in "stderr" am seeing below error. curl -s -d execute="select+*+from+employee01;" -d statusdir="/user/root/" 'http://localhost:50111/templeton/v1/hive?user.name=root' {"id":"job_1491487510034_0007"} =========================== # hadoop fs -cat stderr WARNING: Use "yarn jar" to launch YARN applications. Logging initialized using configuration in jar:file:/data3/hadoop/yarn/local/usercache/root/filecache/37/hive-common.jar!/hive-log4j.properties Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:494) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1533) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3000) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3019) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:475) ... 8 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1531) ... 14 more Caused by: MetaException(message:Got exception: java.net.URISyntaxException Malformed escape pair at index 9: thrift://%HOSTGROUP::host_group_master1%:9933) at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1223) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:229) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) ... 19 more =================================== # hadoop fs -cat stdout ------> (No output) # Templeton status is ok. curl http://localhost:50111/templeton/v1/status {"version":"v1","status":"ok"} ... View more

## Trying to distcp from insecure hadoop cluster(2.3.4) to secure(2.4.2) hadoop cluster and it's failing with below error "Invalid arguments: SIMPLE authentication is not enabled" ## Can any one confirm, is it possible to distcp from insecure to secure hadoop cluster as i have tried with secure to insecure and it's working fine. And also below link is the hdp distcp matrix and where in "HDP2.x" only from secure to insecure hdfs was successfully but no information about insecure to secure. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_Sys_Admin_Guides/content/ref-cfb69f75-d06f-46a2-862f-efeba959b152.1.html ## If anyone has tested insecure to secure, can you kindly confirm this. And is their any configuration required for insecure to secure distcp. ## And, while doing insecure to secure distcp, it's failing with "SIMPLE authentication is not enabled". I hope it's saying about target cluster(secure) and is their any way to enable simple authentication on secure cluster. ## And also below distcp throws "Invalid arguments" error, any idea why distcp is throwing this error, am i missing anything in the distcp. =========================== hdfs@master02:~> hadoop distcp -Dipc.client.fallback-to-simple-auth-allowed=true hdfs://HDP23:8020/test01.txt hdfs://HDP24:8020/ 17/04/05 00:09:28 ERROR tools.DistCp: Invalid arguments: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2118) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1315) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1311) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1311) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:217) at org.apache.hadoop.tools.DistCp.run(DistCp.java:116) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:430) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client.call(Client.java:1427) at org.apache.hadoop.ipc.Client.call(Client.java:1358) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116) ... 9 more Invalid arguments: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] usage: distcp OPTIONS [source_path...] <target_path> ======================================== ... View more

I have set-up ACL for a yarn queue(q01) from Ambari 'yarn queue manager' to allow only one user(user1) to submit jobs into the Queue. But irrespective of ACL's to the queue, all the users were able to submit jobs to the queue. Kindly let me know, is anything wrong am i doing here or any other configuration I missed. Like below, aCL setup for queue 'q01' and parent queue root for user 'user1' to submit jobs: ========================= <property> <name>yarn.scheduler.capacity.root.acl_submit_applications</name> <value>user1 </value> </property> <property> <name>yarn.scheduler.capacity.root.q01.acl_submit_applications</name> <value>user1 </value> </property> ======================== Scenario 1: As per above ACL to q01, only user1 should be able to submit job but user2 was also able to submit job to q01 in below scenario2. ================================= beeline -u jdbc:hive2://localhost:10000/default -n user1 -p user1 --hiveconf hive.execution.engine=mr 0: jdbc:hive2://localhost:10000/default> set mapred.job.queue.name=q01; No rows affected (0.089 seconds) insert into test_u01 values (1); INFO : Table default.test_u01 stats: [numFiles=42, numRows=42, totalSize=84, rawDataSize=42] No rows affected (21.783 seconds) Scenario 2: ================================== beeline -u jdbc:hive2://localhost:10000/default -n user2 -p user2 --hiveconf hive.execution.engine=mr set mapred.job.queue.name=q01; 0: jdbc:hive2://localhost:10000/default> insert into test_u01 values (1); INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Table default.test_u01 stats: [numFiles=43, numRows=43, totalSize=86, rawDataSize=43] No rows affected (21.616 seconds) =================================== ... View more

After configuring below queue's in yarn and after submitting the jobs to created Queue's, the jobs were failing with below error. ERROR: Failed to submit application_XXXXXX to YARN : Application application_XXXXXX submitted by user to unknown queue: root.q01 ## Queue's created in YARN: ---------------------------------------- hdfs@master01:~> mapred queue -list ====================== Queue Name : default Queue State : stopped Scheduling Info : Capacity: 0.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0 ====================== Queue Name : q01 Queue State : running Scheduling Info : Capacity: 50.0, MaximumCapacity: 60.000004, CurrentCapacity: 0.0 ====================== Queue Name : q02 Queue State : running Scheduling Info : Capacity: 50.0, MaximumCapacity: 50.0, CurrentCapacity: 0.0 ====================== Queue Name : child02 Queue State : running Scheduling Info : Capacity: 100.0, MaximumCapacity: 100.0, CurrentCapacity: 0.0 ----------------------------------------- ## Below are the few scenarios, where jobs failed to submit to Q with unknown Queue. Scenario 1: JOB failed to submit, when Q named was provided by appending with parent root Q name(root.q01). -------------------------------------------- set mapred.job.queue.name=root.q01; insert into test_u01 values (1); Failed to submit application_1470318759626_0046 to YARN : Application application_1470318759626_0046 submitted by user user1 to unknown queue: root.q01 Scenario 2: Job executed successfully, when Q named was provided only with child Q(q01). ====================================== set mapred.job.queue.name=q01; insert into test_u01 values (1); INFO : Table default.test_u01 stats: [numFiles=40, numRows=40, totalSize=80, rawDataSize=40] No rows affected (20.125 seconds) ====================================== Scenario 3: Need to execute job into child of a parent Q and where the parent Q is child of root Q. I would to execute the job in child02 which was created in the above Q's but with below Q name it error. ====================================== set mapred.job.queue.name=q02.child02; insert into test_u01 values (1); Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1470318759626_0049 to YARN : Application application_1470318759626_0049 submitted by user user1 to unknown queue: q02.child02 ===================================== Can anyone, please explain how to use Queue names while executing mapreduce jobs and what is the best source to get the actual Q names. And for above scenario 3, what should be the Q name to execute the job successfully in the child Q. ... View more