Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sentry Kerberos Issue

Solved Go to solution

Sentry Kerberos Issue

Explorer

Hi,

 

I have a CDH 5.1.3 cluster managed by cloudera manager. It's fully kerberised and uses AD 2003 as the KDC. I have enabled sentry and everything seems to work fine. However, every morning, beeline stops working and the hiveserver2 log shows the below issue. Currently the only fix i have is to restart hiveserver. With sentry disabled everything works fine. I've tried redeploying kerberos configs, regenerating credentials and increasing the kerberos ticket timeout. Anyone have any ideas where to look?

 

8:34:40.148 AMERRORsentry.org.ape.thrift.transport.TSaslTransport
SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at sentry.org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at sentry.org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:1)
at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient$UgiSaslClientTransport.baseOpen(SentryPolicyServiceClient.java:115)
at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient$UgiSaslClientTransport.access$000(SentryPolicyServiceClient.java:77)
at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient$UgiSaslClientTransport$1.run(SentryPolicyServiceClient.java:101)
at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient$UgiSaslClientTransport$1.run(SentryPolicyServiceClient.java:99)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient$UgiSaslClientTransport.open(SentryPolicyServiceClient.java:99)
at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient.<init>(SentryPolicyServiceClient.java:151)
at org.apache.sentry.provider.db.SimpleDBProviderBackend.<init>(SimpleDBProviderBackend.java:52)
at org.apache.sentry.provider.db.SimpleDBProviderBackend.<init>(SimpleDBProviderBackend.java:48)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.getAuthProvider(HiveAuthzBinding.java:247)
at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.<init>(HiveAuthzBinding.java:88)
at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.<init>(HiveAuthzBinding.java:81)
at org.apache.sentry.binding.hive.HiveAuthzBindingHook.<init>(HiveAuthzBindingHook.java:98)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:374)
at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:59)
at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1162)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:440)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:352)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:995)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:988)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:98)
at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:163)
at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:222)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:204)
at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:168)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:316)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1373)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1358)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 49 more
8:34:40.151 AMWARNorg.apache.hadoop.security.UserGroupInformation
PriviledgedActionException as:hive/gb-slo-hdp-0001.dunnhumby.co.uk@DUNNHUMBY.CO.UK (auth:KERBEROS) cause:sentry.org.apache.thrift.transport.TTransportException: GSS initiate failed
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Sentry Kerberos Issue

Hi,

This is a known issue in Sentry that was fixed in CDH 5.1.4 and 5.2. Please upgrade your Cloudera Manager, then upgrade CDH to resolve this issue.

Be sure to follow upgrade instructions, as there are special steps required when upgrading to CDH 5.2.
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/installation_upgrade.ht...

Thanks,
Darren
3 REPLIES 3
Highlighted

Re: Sentry Kerberos Issue

Hi,

This is a known issue in Sentry that was fixed in CDH 5.1.4 and 5.2. Please upgrade your Cloudera Manager, then upgrade CDH to resolve this issue.

Be sure to follow upgrade instructions, as there are special steps required when upgrading to CDH 5.2.
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/installation_upgrade.ht...

Thanks,
Darren

Re: Sentry Kerberos Issue

Explorer

Many thanks Darren!

Re: Sentry Kerberos Issue

Explorer

I'm experiencing the same issue in 5.7.0. Can you offer any detail on the issue and why we might be sseing it in 5.7.0?