Hive, Sentry, Kerberos and Active Directory - "No groups found for user XXXXXX"

[#1]

Hi.

My company is running a CDH Cluster, with Hue setup with AD. Sentry and Hive. Below all this we also have Kerberos.

The main problem right now is that when Hive tries to search for the groups of a user I get this error.

2017-12-21 14:12:57,687 WARN org.apache.hadoop.security.LdapGroupsMapping: [HiveServer2-Handler-Pool: Thread-108]: Failed to get groups for user ex76196 (retry=0) by javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C09042A, comment: AcceptSecurityContext error, data 52e, v3839]
2017-12-21 14:12:57,706 WARN org.apache.hadoop.security.LdapGroupsMapping: [HiveServer2-Handler-Pool: Thread-108]: Failed to get groups for user ex76196 (retry=1) by javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C09042A, comment: AcceptSecurityContext error, data 52e, v3839]
2017-12-21 14:12:57,724 WARN org.apache.hadoop.security.LdapGroupsMapping: [HiveServer2-Handler-Pool: Thread-108]: Failed to get groups for user ex76196 (retry=2) by javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C09042A, comment: AcceptSecurityContext error, data 52e, v3839]
2017-12-21 14:12:57,724 WARN org.apache.sentry.provider.common.HadoopGroupMappingService: [HiveServer2-Handler-Pool: Thread-108]: Unable to obtain groups for ex76196
java.io.IOException: No groups found for user ex76196
at org.apache.hadoop.security.Groups.noGroupsForUser(Groups.java:190)
at org.apache.hadoop.security.Groups.access$400(Groups.java:69)
at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:307)
at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:257)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:215)
at org.apache.sentry.provider.common.HadoopGroupMappingService.getGroups(HadoopGroupMappingService.java:60)
at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.getGroups(HiveAuthzBinding.java:372)
at org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:395)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:449)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:312)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1201)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1188)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:143)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:215)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:326)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:425)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:402)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:500)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

When I try to setup Hive with AD/LDAP it says that only Kerberos or AD/LDAP can be on. Anyone have any idea how to solve this?

The objective is basically give the AD groups permissions to the Hive tables. 

Kind of lost right now... any ideas would be very appreciated.

Thanks.

[#2]

Even i am having same issue. Could some help on this

[#3]

I'm currently trying to run HDFS DFS -LS / with 1 kerberos principal (one that should be in the AD) and we are having some issues... try to run debug on kerberos to check if you can run commands on HDFS with a principal that is in the AD.

We are doing this in order to test if the problem is in the HDFS/Kerberos/AD configuration.

[#4]

@CTSEH1,

Please confirm that you are seeing exactly the same LDAP problem.  "No groups found for user..." errors can have many causes.  We would need to see logs leading up to and including the error in order to understand if we are seeing exactly the same cuase.

Ben

[#5]

Any resolution to this issue?

I am currently encountering the same issue. My setup is CDH5.15 with Oracle Directory Server.

All the groups and users described below were created in LDAP. All the groups and users show up in Hue as expected. But each operation in either Hive or Impala (through HUE) ends up in a "no groups for ...." error in the sentry log.

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/cm_sg_cm_users_principals.html

[#6]

I don't fully remember but, I think back then we had to use SSSD (Via LDAP. We have another customer via Keytab) to fetch the groups with hadoop.security.group.mapping = org.apache.hadoop.security.ShellBasedUnixGroupsMapping

That way we are able to fetch the groups of each user on the backend.

[#7]

@JoaoBarreto,

Based on the stack trace and errors, you have HDFS configured for LDAP Group Mapping which means hadoop applications will resolve group membership via LDAP.   The LDAP configuration is in your HDFS configuration.

This group lookup is outside of kerberos completely.

We see that the LDAP connection fails with "error code 49".

This means that the Bind DN and Bind DN Password provided in the Cloudera Manager HDFS configuration for LDAP Group Mapping do not match what is in the LDAP server you have configured for those group lookups.

Since the client cannot lookup groups, the group is not found and the operation fails with the error.

To correct, confirm with your LDAP administrator that the user and password you have configured are correct.

it is possible that the Active Directory user account you were using had its password changed if this configuration worked at some time in the past.

[#8]

I encountered the same error but despite the fact I set group mapping to LDAP in HDFS group mapping with appropriate bind user ,in the log of Sentry I'm getting a warning with ShellBasedUnixGroupsMapping and not LdapGroupsMapping. Need a help with ASAP.