Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hiverserver2 logs showing Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

Hiverserver2 logs showing Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

Rising Star

In hiveserver2 logs full of these errors even though service is accessible. Ambari shows Hive is up and running without any alerts. What is causing this error to be generated and how to remove these.

Log stack trace:

2016-09-25 00:00:17,210 INFO retry.RetryInvocationHandler (RetryInvocationHandler.java:invoke(144)) - Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over tph01kdc/39.7.48.2:8020 after 3 fail over attempts. Trying to fail over immediately. java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "TPH01KDC/39.7.48.2"; destination host is: "tph01kdc":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773) at org.apache.hadoop.ipc.Client.call(Client.java:1431) at org.apache.hadoop.ipc.Client.call(Client.java:1358) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1315) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1311) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1311) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:226) at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:123) at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:890) at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:838) at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759) at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637) at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:685) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:648) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:735) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493) at org.apache.hadoop.ipc.Client.call(Client.java:1397) ... 27 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558) at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:373) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:722) ... 30 more Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 39 more

4 REPLIES 4

Re: Hiverserver2 logs showing Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

Guru

Hello @Anshul Sisodia,

From the error stack, looks like HiveServer2's Ranger plugin is trying to write audit log to HDFS & silently failing. If you are not planning to store Hive audit logs in HDFS, you can turn this off. Else we can look into the Hive Ranger audit configuration.

This error usually means that Ranger audit plugin does not have a Kerberos ticket to write log records to HDFS. Please check if the Ranger audit configuration (audit user principal & keytab) is configured & working properly.

Hope this helps.

Highlighted

Re: Hiverserver2 logs showing Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

New Contributor

I am seeing the same issue, but it does not seem related to the principal. It looks like the RANGER-1136 Jira: https://issues.apache.org/jira/browse/RANGER-1136

We are watching things now, but it seems that the errors begin when the TGT is either not renewed or another is not obtained. The Jira above has been patched. We're running HDP 2.3.4 in the cluster in question, soon to upgrade to 2.5.3. Will have to check if that bug is fixed in 2.5.3.

Re: Hiverserver2 logs showing Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

New Contributor

@Vipin Rathor

"Please check if the Ranger audit configuration (audit user principal & keytab) is configured & working properly."

Where to check for that? I see where the principal and keytab settings are for hiveserver2 and the metastore, but not specifically for the Ranger plugin.

Re: Hiverserver2 logs showing Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

New Contributor

Hello, we are getting totally same problem with HDP2.6.2.

Could someone confirm https://issues.apache.org/jira/browse/RANGER-1136 is included???

I've checked source, but I couldn't...