Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kerberos integration with KDC leads to GSSException

Highlighted

Kerberos integration with KDC leads to GSSException

Explorer
Hi, 
 
We have a problem with Kerberos ticket from Active directory.
 
SETUP
Our setup is the following:
[Active Directory, realm=ADREALM]
         |
[KDC, realm=LOCALREALM]
         |
[CLOUDERA CLUSTER]
 
As you can see, we there are 2 realms:
  • LOCALREALM: "local" KDC where are all Cloudera cluster principals for users like hdfs, hbase etc.
  • ADREALM: the "remote" KDC of the Active Directory.
 
For testing the integration, we have 2 principals that both resolve to the local user "TEST_USER":
  • TEST_USER@LOCALREALM
  • TEST_USER@adREALM

 

PROBLEM
Basically, if we kinit against local KDC with TEST_USER@LOCALREALM, we are able to perform any expected action, such as 
hdfs dsf -ls / 
We've been working a while with the local KDC on whole platform without any problems. Recently we added the AD integration; the local KDC keeps working as expected.
 
But, if we kinit with TEST_USER@ADREALM, we obtain a ticket, and it seems ok, but we are unable to perform any command on hadoop. I insert a screenshot with the GSSException:
 
[root@master1 ~]# kdestroy
[root@master1 ~]# kinit TEST_USER@ADREALM
Password for TEST_USER@ADREALM: 
[root@master1 ~]# klist -e
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: TEST_USER@ADREALM

Valid starting     Expires            Service principal
05/02/19 15:27:48  06/02/19 01:27:48  krbtgt/ADREALM@ADREALM
	renew until 12/02/19 15:27:43, Etype (skey, tkt): aes128-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 
[root@master1 ~]# hdfs dfs -ls /
19/02/05 15:27:59 WARN security.UserGroupInformation: PriviledgedActionException as:TEST_USER@ADREALM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
19/02/05 15:28:01 WARN security.UserGroupInformation: PriviledgedActionException as:TEST_USER@ADREALM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
19/02/05 15:28:01 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1549376879231
19/02/05 15:28:06 WARN security.UserGroupInformation: PriviledgedActionException as:TEST_USER@ADREALM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
19/02/05 15:28:06 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1549376879231
Related to the error above, we see this message in the file /var/log/krb5kdc.log at the master1.clouderacluster.net (machine of the local KDC):
 
Feb 05 15:31:49 master1.clouderacluster.net krb5kdc[18098](info): TGS_REQ (3 etypes {17 23 16}) 10.251.188.10: PROCESS_TGS: authtime 0,  <unknown client> for hdfs/master1.clouderacluster.net@LOCALREALM, Decrypt integrity check failed
We suspect that it might be related a the encryption mismatch, because if we compare the output of klist  -e, the difference is in the tkt ecryption section. Where AD gives 256, local ticket gives 128:
Etype (skey, tkt): aes128-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96 
 
The contents of the krb5.conf is:
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 rc4-hmac des3-cbc-sha1 des-cbc-md5
 default_tkt_enctypes = aes128-cts-hmac-sha1-96 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 rc4-hmac des3-cbc-sha1 des-cbc-md5
 default_tgs_enctypes = aes128-cts-hmac-sha1-96 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 rc4-hmac des3-cbc-sha1 des-cbc-md5
  
 default_realm = LOCALREALM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 allow-weak-crypto = true
 udp_preference_limit = 1

[realms]
 LOCALREALM = {
  kdc = master1.clouderacluster.net
  admin_server = master1.clouderacluster.net
  max_renewable_life = 7d 0h 0m 0s
  default_principal_flags = +renewable
 }
 
 ADREALM = {
  kdc = ad.domain.net:88
  admin_server = ad.domain.net:749
 }

[domain_realm]
 .localrealm = LOCALREALM
 localrealm = LOCALREALM
 .ad.domain.net = ADREALM
 ad.domain.net = ADREALM
I repeat, this happens only with the AD ticket.
All JDKs have JCE installed. 
Thanks in advance for any suggestions!
 
2 REPLIES 2

Re: Kerberos integration with KDC leads to GSSException

This is probably too simple, but I noticed that you ran kinit as TEST_USER, but then used root to run the command against HDFS.

Re: Kerberos integration with KDC leads to GSSException

New Contributor

Nice Explanation......

Don't have an account?
Coming from Hortonworks? Activate your account here