Setting quota will work. Queries will fail with quota errors.  ... View more

Hello Harsh,   Thank you for your reply. I was able to narrow down the cause. It was due to the membership of this user account in a specific group. Once we removed the user from that group the issue got resolved.  ... View more

Any updates?  ... View more

- Do any of the outputs in the groups command you run return pure numeric results, instead of actual string names? No. - What's the exit code after you execute 'id -gn username' for the affected user? You may run 'echo $?' to grab exit code after the command.   $ id -gn user ; echo $? 1 $ - Please paste the full stack trace, which should include a trace of an IOException after the log message as an underlying 'Caused by'. This would explain the reason behind why the partial group resolution further fails. +++++++ 2018-08-07 15:17:35,638 WARN org.apache.sentry.provider.common.HadoopGroupMappingService: [HiveServer2-Handler-Pool: Thread-2934561]: Unable to obtain groups for <user> java.io.IOException: No groups found for user <user> at org.apache.hadoop.security.Groups.noGroupsForUser(Groups.java:197) at org.apache.hadoop.security.Groups.getGroups(Groups.java:220) at org.apache.sentry.provider.common.HadoopGroupMappingService.getGroups(HadoopGroupMappingService.java:60) at org.apache.sentry.provider.common.ResourceAuthorizationProvider.getGroups(ResourceAuthorizationProvider.java:167) at org.apache.sentry.provider.common.ResourceAuthorizationProvider.doHasAccess(ResourceAuthorizationProvider.java:97) at org.apache.sentry.provider.common.ResourceAuthorizationProvider.hasAccess(ResourceAuthorizationProvider.java:91) at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.authorize(HiveAuthzBinding.java:319) at org.apache.sentry.binding.hive.HiveAuthzBindingHook.filterShowDatabases(HiveAuthzBindingHook.java:907) at org.apache.sentry.binding.metastore.SentryMetaStoreFilterHook.filterDb(SentryMetaStoreFilterHook.java:131) at org.apache.sentry.binding.metastore.SentryMetaStoreFilterHook.filterDatabases(SentryMetaStoreFilterHook.java:59) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:1042) at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:105) at com.sun.proxy.$Proxy19.getDatabases(Unknown Source) at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2034) at com.sun.proxy.$Proxy19.getDatabases(Unknown Source) at org.apache.hive.service.cli.operation.GetSchemasOperation.runInternal(GetSchemasOperation.java:59) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:337) at org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:503) at org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:320) at org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:546) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1373) at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1358) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) +++++++ - Is there any particular difference to this username vs. others? For ex., does it start with a special character instead of alpha-num, etc.?   Normal user account.  ... View more

INFO - 2018-07-24 05:16:24,511 INFO [main] retry.RetryInvocationHandler (RetryInvocationHandler.java:invoke(148)) - Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over namenode_host/IP after 1 fail over attempts. Trying to fail over immediately. INFO - java.net.BindException: Problem binding to [scheduler_hostname/IP:0] java.net.BindException: Cannot assign requested address; For more details see: http://wiki.apache.org/hadoop/BindException INFO - at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) INFO - at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) INFO - at java.lang.reflect.Constructor.newInstance(Constructor.java:526) INFO - at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) INFO - at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:720) INFO - at org.apache.hadoop.ipc.Client.call(Client.java:1476) INFO - at org.apache.hadoop.ipc.Client.call(Client.java:1409) INFO - at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) INFO - at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) INFO - at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) INFO - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) INFO - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) INFO - at java.lang.reflect.Method.invoke(Method.java:606) INFO - at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) INFO - at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) INFO - at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source) INFO - at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2123) INFO - at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1253) INFO - at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1249) INFO - at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) INFO - at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1249) INFO - at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1417) INFO - Caused by: java.net.BindException: Cannot assign requested address INFO - at sun.nio.ch.Net.connect0(Native Method) INFO - at sun.nio.ch.Net.connect(Net.java:465) INFO - at sun.nio.ch.Net.connect(Net.java:457) INFO - at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) INFO - at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) INFO - at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530) INFO - at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494) INFO - at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:615) INFO - at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:714) INFO - at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) INFO - at org.apache.hadoop.ipc.Client.getConnection(Client.java:1525) INFO - at org.apache.hadoop.ipc.Client.call(Client.java:1448) INFO - ... 17 more ... View more

Hi,   I am getting the same outputs in my name nodes as well.     #groups <user ID> Returns proper group mapping.   # hdfs groups <user ID> No groups returned.     This is happening only for a specific user account and we are using ShellBasedUnixGroupsMapping.  Sample log: ++++++++ org.apache.hadoop.security.ShellBasedUnixGroupsMapping: unable to return groups for user ID PartialGroupNameException can't execute the shell command to get the list of group id for user 'ID' at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:228) +++++++     ... View more

Thanks, Harsh for your reply.   I am executing this from gateway node. I am using SSSD and able to fetch right groups using "groups <ID>" command. However, "hdfs groups" is not showing any groups. This is the same when checked from other nodes in the cluster as well. This is happening to only one particular user.  ... View more

Also, if I am using mem_lim query option while running query, will it bypass the "max mem" set at admission control setting? For example, I have set 400GB in max_mem and using mem_lim as 450 while running the query.  Another issue, if mem_lim is set at pool level, the number of queries that can be executed will be reduced right? Since mem_lim amount of RAM will be reserved for each query.  ... View more