question Re: Knox impersonation issue in Support Questions

Knox impersonation issue

vmshah — Mon, 19 Aug 2019 08:26:29 GMT

I have following use case:

Application connecting to Knox gateway and trying to run hive source -> hive target. The way it is transformed is Knox connects to Hive Service and submit the request.

I have user guest created for Knox access and created as Unix user also. I am trying to impersonate user adapqa while submitting job via Knox.

I am getting following error in hiveserver2 log.

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hive is not allowed to impersonate adpqa at org.apache.hadoop.ipc.Client.call(Client.java:1427) at org.apache.hadoop.ipc.Client.call(Client.java:1358) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301) at org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(FileUtils.java:757) at org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(StorageBasedAuthorizationProvider.java:364) at org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(StorageBasedAuthorizationProvider.java:339) ... 74 more 2016-02-12 12:38:17,578 INFO [HiveServer2-HttpHandler-Pool: Thread-36]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(127)) - Could not validate cookie sent, will try to generate a new cookie 2016-02-12 12:38:17,578 INFO [HiveServer2-HttpHandler-Pool: Thread-36]: thrift.ThriftHttpServlet (ThriftHttpServlet.java:doPost(169)) - Cookie added for clientUserName anonymous 2016-02-12 12:38:17,578 INFO [HiveServer2-HttpHandler-Pool: Thread-36]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8 2016-02-12 12:38:17,580 INFO [HiveServer2-HttpHandler-Pool: Thread-36]: metastore.ObjectStore (ObjectStore.java:initialize(290)) - ObjectStore, initialize called 2016-02-12 12:38:18,261 WARN [HiveServer2-HttpHandler-Pool: Thread-36]: conf.HiveConf (HiveConf.java:initialize(2774)) - HiveConf of name hive.server2.enable.impersonation does not exist 2016-02-12 12:38:18,262 INFO [HiveServer2-HttpHandler-Pool: Thread-36]: metastore.ObjectStore (ObjectStore.java:getPMF(375)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,Database,Type,FieldSchema,Order"

I have made sure that adpqa does exist as a user both on unix and hdfs.

adpqa@ivlhdp61:/var/log/hive> hadoop fs -ls /user

Found 5 items drwxr-xr-x - adpqa supergroup 0 2016-02-12 11:54 /user/adpqa

Both hive and adpqa are part of users group

adpqa@ivlhdp61:/var/log/hive> groups hive

hive : hadoop users

adpqa@ivlhdp61:/var/log/hive> groups adpqa

adpqa : users dialout video hadoop

Following is HDFS configuration on cluster.

I am unable to understand why do we get error "User: hive is not allowed to impersonate adpqa".

Is that some more configuration missing?

Re: Knox impersonation issue

nsabharwal — Fri, 12 Feb 2016 18:57:32 GMT

@Vishal Dhavale

Try with * and see if it works.

Re: Knox impersonation issue

vmshah — Fri, 12 Feb 2016 22:07:46 GMT

Can you explain for which property we need to use *.

Actually I have tried * for couple of properties for which there was specific user mentioned. It will help if i can identify due to which i am facing this issue as It is not clear from error message.

Re: Knox impersonation issue

nsabharwal — Fri, 12 Feb 2016 22:13:11 GMT

@Vishal Shah I don't want to waste your time 🙂 as I am little confuse by your settings. I do understand and I do see that you have create proxy group for adpqa ...I am curious to see the behavior if you make all settings * for adpqa or remove proxy setting for adpqa

Re: Knox impersonation issue

vmshah — Fri, 12 Feb 2016 22:16:21 GMT

Alright. I would give that shot as well. Meanwhile i will try to figure out the culprit property.

Thanks

Re: Knox impersonation issue

kevin_minder — Fri, 12 Feb 2016 22:23:15 GMT

Looks like you are missing

hadoop.proxyuser.knox.groups=users
hadoop.proxyuser.knox.hosts=*

Also note that you should probably not have

hadoop.proxyuser.guest.groups=users
hadoop.proxyuser.guest.hosts=*

as this is essentially saying that the 'guest' user is allowed to impersonate anyone in the 'users' group.

Beyond that you need to ensure that your user 'adpqa' is in group 'users'.

Re: Knox impersonation issue

nsabharwal — Fri, 12 Feb 2016 22:31:33 GMT

@Vishal Shah

See this ...Kevin is Knox expert. Thanks @Kevin Minder

Re: Knox impersonation issue

vmshah — Mon, 15 Feb 2016 17:21:30 GMT

Hi Kevin,

After adding following properties issue is still same.

hadoop.proxyuser.knox.groups=users

hadoop.proxyuser.knox.hosts=*

Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: MetaException(message:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hive is not allowed to impersonate adpqa) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:1386) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.getDatabase(BaseSemanticAnalyzer.java:1378) at

On cluster i have made sure adpqa is group users

adpqa@ivlhdp61:/var/log/hive> groups adpqa

adpqa : users dialout video hadoop

adpqa@ivlhdp61:/var/log/hive> groups hive

hive : hadoop users

One more thing i had found was following from hiveserver2 log.

2016-02-15 14:20:18,497 WARN [HiveServer2-HttpHandler-Pool: Thread-39]: conf.HiveConf (HiveConf.java:initialize(2774)) - HiveConf of name hive.server2.enable.impersonation does not exist

Although property is set to true for Hive Service in Ambari. Not sure if this is the actual reason for impersonation not working while connecting to HiveServer2 via Knox.

Re: Knox impersonation issue

pminovic — Tue, 16 Feb 2016 14:01:42 GMT

Hi @Vishal Shah, you also need, in Hive-->Configs:

webhcat.proxyuser.knox.groups=*
webhcat.proxyuser.knox.hosts=*
hive.server2.allow.user.substitution=true

Try knox.groups and hosts first with "*" and if it works reduce permissions to for example "users" and your KNOX host FQDN. Full manual here, scroll down to the Hive section.

Re: Knox impersonation issue

vmshah — Tue, 16 Feb 2016 18:43:17 GMT

Hi Predrag,

We tried this as well. But issue still exist.

It is strange that using beeline with same jdbc connection string i am able to execute queries successfully.

But when running from an application it does not work.

Re: Knox impersonation issue

nsabharwal — Mon, 19 Aug 2019 08:26:16 GMT

@Vishal Shah Your connection works fine without knox?

Is this set to true?

Re: Knox impersonation issue

kevin_minder — Tue, 16 Feb 2016 22:49:56 GMT

I didn't understand that beeline was working via Knox already. A few questions then:

What application is making the HS2 call via Knox?
Is the application using JDBC or ODBC drivers and what version?
What does your JDBC connect string look like (without real hostname or passwords of course)?

Re: Knox impersonation issue

vmshah — Fri, 26 Feb 2016 14:42:23 GMT

Hi Kevin,

Thanks for the reply. I was away for a while couldn't follow up on the issue.

With the new cluster setup, we do not see this issue anymore. I believe issue was due to improper configuration.

Re: Knox impersonation issue

vmshah — Fri, 26 Feb 2016 14:42:40 GMT

With the new cluster setup, we do not see this issue anymore. I believe issue was due to improper configuration.

Re: Knox impersonation issue

melba_swapana — Fri, 06 Jan 2017 07:44:12 GMT

we are having the same issue with and our HDP version is 2.4.2, here are all the setting we have already implemented. Our beeline works for all users. There is no permission issue either.

Here is the error logs and I have already attached few settings from our environment.

2017-01-03 10:04:22,851 INFO [HiveServer2-Handler-Pool: Thread-67181]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V1 2017-01-03 10:04:22,854 WARN [HiveServer2-Handler-Pool: Thread-67181]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(308)) - Error opening session: org.apache.hive.service.cli.HiveSQLException: Failed to validate proxy privilege of tabsrvtest for btaylo at org.apache.hive.service.auth.HiveAuthFactory.verifyProxyAccess(HiveAuthFactory.java:379) at org.apache.hive.service.cli.thrift.ThriftCLIService.getProxyUser(ThriftCLIService.java:731) at org.apache.hive.service.cli.thrift.ThriftCLIService.getUserName(ThriftCLIService.java:367) at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:394) at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:297) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1257) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1242) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User: tabsrvtest is not allowed to impersonate

btaylo

at org.apache.hadoop.security.authorize.DefaultImpersonationProvider.authorize(DefaultImpersonationProvider.java:119) at org.apache.hadoop.security.authorize.ProxyUsers.authorize(ProxyUsers.java:102) at org.apache.hadoop.security.authorize.ProxyUsers.authorize(ProxyUsers.java:116) at org.apache.hive.service.auth.HiveAuthFactory.verifyProxyAccess(HiveAuthFactory.java:375) ... 13 more

2017-01-03 10:04:22,866 WARN [HiveServer2-Handler-Pool: Thread-67181]: thrift.ThriftCLIService (ThriftCLIService.java:CloseSession(456)) - Error closing session: java.nio.BufferUnderflowException at java.nio.Buffer.nextGetIndex(Buffer.java:506) at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412) at org.apache.hive.service.cli.HandleIdentifier.<init>(HandleIdentifier.java:46) at org.apache.hive.service.cli.Handle.<init>(Handle.java:38) at org.apache.hive.service.cli.SessionHandle.<init>(SessionHandle.java:45) at org.apache.hive.service.cli.SessionHandle.<init>(SessionHandle.java:41) at org.apache.hive.service.cli.thrift.ThriftCLIService.CloseSession(ThriftCLIService.java:447) at org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1277) at org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1262) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

core-site.png hive-settings.png