Knox impersonation issue

[#1]

I have following use case:

Application connecting to Knox gateway and trying to run hive source -> hive target. The way it is transformed is Knox connects to Hive Service and submit the request.

I have user guest created for Knox access and created as Unix user also. I am trying to impersonate user adapqa while submitting job via Knox.

I am getting following error in hiveserver2 log.

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hive is not allowed to impersonate adpqa at org.apache.hadoop.ipc.Client.call(Client.java:1427) at org.apache.hadoop.ipc.Client.call(Client.java:1358) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301) at org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(FileUtils.java:757) at org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(StorageBasedAuthorizationProvider.java:364) at org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(StorageBasedAuthorizationProvider.java:339) ... 74 more 2016-02-12 12:38:17,578 INFO [HiveServer2-HttpHandler-Pool: Thread-36]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(127)) - Could not validate cookie sent, will try to generate a new cookie 2016-02-12 12:38:17,578 INFO [HiveServer2-HttpHandler-Pool: Thread-36]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(169)) - Cookie added for clientUserName anonymous 2016-02-12 12:38:17,578 INFO [HiveServer2-HttpHandler-Pool: Thread-36]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8 2016-02-12 12:38:17,580 INFO [HiveServer2-HttpHandler-Pool: Thread-36]: metastore.ObjectStore (ObjectStore.java:initialize(290)) - ObjectStore, initialize called 2016-02-12 12:38:18,261 WARN [HiveServer2-HttpHandler-Pool: Thread-36]: conf.HiveConf (HiveConf.java:initialize(2774)) - HiveConf of name hive.server2.enable.impersonation does not exist 2016-02-12 12:38:18,262 INFO [HiveServer2-HttpHandler-Pool: Thread-36]: metastore.ObjectStore (ObjectStore.java:getPMF(375)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,Database,Type,FieldSchema,Order"

I have made sure that adpqa does exist as a user both on unix and hdfs.

adpqa@ivlhdp61:/var/log/hive> hadoop fs -ls /user

Found 5 items drwxr-xr-x - adpqa supergroup 0 2016-02-12 11:54 /user/adpqa

Both hive and adpqa are part of users group

adpqa@ivlhdp61:/var/log/hive> groups hive

hive : hadoop users

adpqa@ivlhdp61:/var/log/hive> groups adpqa

adpqa : users dialout video hadoop

Following is HDFS configuration on cluster.

I am unable to understand why do we get error "User: hive is not allowed to impersonate adpqa".

Is that some more configuration missing?

[#2]

@Vishal Dhavale

Try with * and see if it works.

[#3]

Can you explain for which property we need to use *.

Actually I have tried * for couple of properties for which there was specific user mentioned. It will help if i can identify due to which i am facing this issue as It is not clear from error message.

[#4]

@Vishal Shah I don't want to waste your time 🙂 as I am little confuse by your settings. I do understand and I do see that you have create proxy group for adpqa ...I am curious to see the behavior if you make all settings * for adpqa or remove proxy setting for adpqa

[#5]

Alright. I would give that shot as well. Meanwhile i will try to figure out the culprit property.

Thanks

[#6]

@Vishal Shah Your connection works fine without knox?

Is this set to true?

[#7]

we are having the same issue with and our HDP version is 2.4.2, here are all the setting we have already implemented. Our beeline works for all users. There is no permission issue either.

Here is the error logs and I have already attached few settings from our environment.

2017-01-03 10:04:22,851 INFO [HiveServer2-Handler-Pool: Thread-67181]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V1 2017-01-03 10:04:22,854 WARN [HiveServer2-Handler-Pool: Thread-67181]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(308)) - Error opening session: org.apache.hive.service.cli.HiveSQLException: Failed to validate proxy privilege of tabsrvtest for btaylo at org.apache.hive.service.auth.HiveAuthFactory.verifyProxyAccess(HiveAuthFactory.java:379) at org.apache.hive.service.cli.thrift.ThriftCLIService.getProxyUser(ThriftCLIService.java:731) at org.apache.hive.service.cli.thrift.ThriftCLIService.getUserName(ThriftCLIService.java:367) at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:394) at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:297) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1257) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1242) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User: tabsrvtest is not allowed to impersonate

btaylo

at org.apache.hadoop.security.authorize.DefaultImpersonationProvider.authorize(DefaultImpersonationProvider.java:119) at org.apache.hadoop.security.authorize.ProxyUsers.authorize(ProxyUsers.java:102) at org.apache.hadoop.security.authorize.ProxyUsers.authorize(ProxyUsers.java:116) at org.apache.hive.service.auth.HiveAuthFactory.verifyProxyAccess(HiveAuthFactory.java:375) ... 13 more

2017-01-03 10:04:22,866 WARN [HiveServer2-Handler-Pool: Thread-67181]: thrift.ThriftCLIService (ThriftCLIService.java:CloseSession(456)) - Error closing session: java.nio.BufferUnderflowException at java.nio.Buffer.nextGetIndex(Buffer.java:506) at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412) at org.apache.hive.service.cli.HandleIdentifier.<init>(HandleIdentifier.java:46) at org.apache.hive.service.cli.Handle.<init>(Handle.java:38) at org.apache.hive.service.cli.SessionHandle.<init>(SessionHandle.java:45) at org.apache.hive.service.cli.SessionHandle.<init>(SessionHandle.java:41) at org.apache.hive.service.cli.thrift.ThriftCLIService.CloseSession(ThriftCLIService.java:447) at org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1277) at org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1262) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

core-site.pnghive-settings.png

[#8]

Looks like you are missing

hadoop.proxyuser.knox.groups=users
hadoop.proxyuser.knox.hosts=*

Also note that you should probably not have

hadoop.proxyuser.guest.groups=users
hadoop.proxyuser.guest.hosts=*

as this is essentially saying that the 'guest' user is allowed to impersonate anyone in the 'users' group.

Beyond that you need to ensure that your user 'adpqa' is in group 'users'.

[#9]

@Vishal Shah

See this ...Kevin is Knox expert. Thanks @Kevin Minder

[#10]

Hi Kevin,

After adding following properties issue is still same.

hadoop.proxyuser.knox.groups=users

hadoop.proxyuser.knox.hosts=*

Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: MetaException(message:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hive is not allowed to impersonate adpqa) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:1386) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:1378) at

On cluster i have made sure adpqa is group users

adpqa@ivlhdp61:/var/log/hive> groups adpqa

adpqa : users dialout video hadoop

adpqa@ivlhdp61:/var/log/hive> groups hive

hive : hadoop users

One more thing i had found was following from hiveserver2 log.

2016-02-15 14:20:18,497 WARN [HiveServer2-HttpHandler-Pool: Thread-39]: conf.HiveConf (HiveConf.java:initialize(2774)) - HiveConf of name hive.server2.enable.impersonation does not exist

Although property is set to true for Hive Service in Ambari. Not sure if this is the actual reason for impersonation not working while connecting to HiveServer2 via Knox.