Alternate days why do i see GSSException: No valid credentials provided?

[#1]

As part of this article we will see that sometimes we get the following very common error while accessing some services (in the following case while running very basic "hdfs" commands).

While running the command we see the error as following:

[root@kjss1 ~]# su - hdfs
[hdfs@kjss1 ~]$ hdfs dfs -ls /user/admin

16/12/16 12:28:27 WARN ipc.Client: Exception encountered while connecting to the server : 
16/12/16 12:28:27 WARN retry.RetryInvocationHandler: Exception while invoking GetUserMappingsProtocolClientSideTranslatorPB.getGroupsForUser over kjss2.example.com/10.20.169.238:8020. Not retrying because failovers (15) exceeded maximum allowed (15)
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "kjss1.example.com/172.26.69.237"; destination host is: "kjss2.example.com":8020; 
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
	at org.apache.hadoop.ipc.Client.call(Client.java:1496)
	at org.apache.hadoop.ipc.Client.call(Client.java:1396)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
	at com.sun.proxy.$Proxy8.getGroupsForUser(Unknown Source)
	at org.apache.hadoop.tools.protocolPB.GetUserMappingsProtocolClientSideTranslatorPB.getGroupsForUser(GetUserMappingsProtocolClientSideTranslatorPB.java:57)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
	at com.sun.proxy.$Proxy9.getGroupsForUser(Unknown Source)
	at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
	at org.apache.hadoop.hdfs.tools.GetGroups.main(GetGroups.java:96)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:720)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:683)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:770)
	at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
	at org.apache.hadoop.ipc.Client.call(Client.java:1449)
	... 16 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757)
	... 19 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
	... 28 more

.

In such cases we will need to first check if the current user who is running the hdfs commands is having a valid kerberos ticket or not. If not then we will need to get the ticket as following:

1). Sudo to the "hdfs" user or whatever user you want to choose who is getting the mentioned error.

[root@kjss1 ~]# su - hdfs

.

2). Do a "klist" to see if you have a valid kerberos ticket or not?

[hdfs@kjss1 ~]$ klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_504)

.

3). If you do not see any valid ticket then list the keytab. To findout what is your "Principal name" ?

[hdfs@kjss1 ~]$ klist -kte /etc/security/keytabs/hdfs.headless.keytab 

Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   4 08/10/16 13:43:31 hdfs-JoyCluster@EXAMPLE.COM (des3-cbc-sha1) 
   4 08/10/16 13:43:31 hdfs-JoyCluster@EXAMPLE.COM (arcfour-hmac) 
   4 08/10/16 13:43:31 hdfs-JoyCluster@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 
   4 08/10/16 13:43:31 hdfs-JoyCluster@EXAMPLE.COM (des-cbc-md5) 
   4 08/10/16 13:43:31 hdfs-JoyCluster@EXAMPLE.COM (aes256-cts-hmac-sha1-96) 

.

4). Now you can do the "kinit" on that Principal name.

[hdfs@kjss1 ~]$ kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-JoyCluster@EXAMPLE.COM 

[hdfs@kjss1 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_504
Default principal: hdfs-JoyCluster@EXAMPLE.COM

Valid starting     Expires            Service principal
12/16/16 12:21:27  12/17/16 12:21:27  krbtgt/EXAMPLE.COM@EXAMPLE.COM
	renew until 12/16/16 12:21:27

.

5). Now try the hdfs commands.

$ hdfs dfs -ls /user
Found 10 items
drwxr-xr-x   - 1hive     hdfs            0 2016-09-23 14:09 /user/1hive
drwxr-xr-x   - admin     hadoop          0 2016-09-08 10:15 /user/admin
drwxrwx---   - ambari-qa hdfs            0 2016-09-08 06:48 /user/ambari-qa
drwxr-xr-x   - admin     hadoop          0 2016-12-01 15:13 /user/ambari-server-joycluster
drwxr-xr-x   - hbase     hdfs            0 2016-08-22 09:49 /user/hbase
drwxr-xr-x   - hcat      hdfs            0 2016-08-10 11:31 /user/hcat
drwxr-xr-x   - hive      hdfs            0 2016-08-10 11:31 /user/hive
drwxr-xr-x   - joy       hadoop          0 2016-12-01 15:25 /user/joy
drwxrwxr-x   - spark     hdfs            0 2016-10-27 05:56 /user/spark
drwxr-xr-x   - zeppelin  hdfs            0 2016-10-27 09:40 /user/zeppelin

Some Additional info on Ticket Expiration:

- Run a "ps -ef | grep $PID" on your commands to see it's JVM arguments. Find out which file is mentioned for the property "-Djava.security.krb5.conf". You will see the "/PATH/TO/krb5.conf" file. Inside that file we will see following kind of settings in it.

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = EXAMPLE.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

[domain_realm]
  .example.com = EXAMPLE.COM
  example.com = EXAMPLE.COM

[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log

[realms]
  EXAMPLE.COM = {
    admin_server = kjss1.example.com
    kdc = kjss1.example.com
  }

.

NOTICE: "ticket_lifetime" is set to 24h (Time duration string.) Sets the default lifetime for initial ticket requests. The default value is 1 day. After that period the ticket will be automatically expire hence we will need to renew the ticket.

Kerberos Java Debug Flags:

A debugging mode can be enabled by setting the system property sun.security.krb5.debug to "true". This setting allows you to follow the program's execution of the Kerberos V5 protocol.