Got GSSException: No valid credentials provided with a valid kerberos ticket

[#1]

I am using CM API installing a CDH cluster on AWS with MIT KDC and JDK 1.8u121. From CM UI, keberos is working fine. I checked CM kerberos encryption types and they match those defined in kdc.conf.

$ sudo cat /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88

[realms]
AWS = {
#master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
}

$ klist -ef

Ticket cache: KEYRING:persistent:1000:1000
Default principal: wzhu@AWS

Valid starting Expires Service principal
03/30/2017 00:20:37 03/31/2017 00:20:37 krbtgt/AWS@AWS
Flags: FI, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

$ hdfs dfs -ls /

...

ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "ip-10-1-30-107.us-west-1.compute.internal/10.1.30.107"; destination host is: "ip-10-1-30-107.us-west-1.compute.internal":8020;
...skipping...
at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:375)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:726)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)

[#2]

P.S. I am on MIT KRB 1.14 version.

[#3]

It seems that JDK was not able to load kerberos ticket from cache.

$ export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true $HADOOP_OPTS"
$ hdfs dfs -ls / 2> /tmp/hdfsls.txt
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
>>>KinitOptions cache name is /tmp/krb5cc_1000

I should see extra KRB debug, but found none.

[#4]

Switched to JDK 1.7 and got the same issue. It seems that JDK can't pick up from the cache.

$ export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true $HADOOP_OPTS"
$ export HADOOP_ROOT_LOGGER=TRACE,console;
$ export HADOOP_JAAS_DEBUG=true

$ hdfs dfs -ls 2> /tmp/hdfsls.txt
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
[UnixLoginModule]: succeeded importing info:
uid = 1000
gid = 1000
supp gid = 4
supp gid = 10
supp gid = 190
supp gid = 1000
Debug is true storeKey false useTicketCache true useKeyTab false doNotPrompt true ticketCache is null isInitiator true KeyTab is null refreshKrb5Config is false principal is null tryFirstPass is false useFirstPass is false storePass is false clearPass is false
Acquire TGT from Cache
>>>KinitOptions cache name is /tmp/krb5cc_1000
Principal is null
null credentials from Ticket Cache
[Krb5LoginModule] authentication failed
Unable to obtain Princpal Name for authentication
[UnixLoginModule]: added UnixPrincipal,
UnixNumericUserPrincipal,
UnixNumericGroupPrincipal(s),
to Subject