Explorer
Posts: 22
Registered: ‎04-28-2016
Kerberos integration with KDC leads to GSSException
[ Edited ]
Hi, 
 
We have a problem with Kerberos ticket from Active directory.
 
SETUP
Our setup is the following:
[Active Directory, realm=ADREALM]
         |
[KDC, realm=LOCALREALM]
         |
[CLOUDERA CLUSTER]
 
As you can see, we there are 2 realms:
  • LOCALREALM: "local" KDC where are all Cloudera cluster principals for users like hdfs, hbase etc.
  • ADREALM: the "remote" KDC of the Active Directory.
 
For testing the integration, we have 2 principals that both resolve to the local user "TEST_USER":
  • TEST_USER@LOCALREALM
  • TEST_USER@adREALM

 

PROBLEM
Basically, if we kinit against local KDC with TEST_USER@LOCALREALM, we are able to perform any expected action, such as 
hdfs dsf -ls / 
We've been working a while with the local KDC on whole platform without any problems. Recently we added the AD integration; the local KDC keeps working as expected.
 
But, if we kinit with TEST_USER@ADREALM, we obtain a ticket, and it seems ok, but we are unable to perform any command on hadoop. I insert a screenshot with the GSSException:
 
[root@master1 ~]# kdestroy
[root@master1 ~]# kinit TEST_USER@ADREALM
Password for TEST_USER@ADREALM: 
[root@master1 ~]# klist -e
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: TEST_USER@ADREALM

Valid starting     Expires            Service principal
05/02/19 15:27:48  06/02/19 01:27:48  krbtgt/ADREALM@ADREALM
	renew until 12/02/19 15:27:43, Etype (skey, tkt): aes128-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 
[root@master1 ~]# hdfs dfs -ls /
19/02/05 15:27:59 WARN security.UserGroupInformation: PriviledgedActionException as:TEST_USER@ADREALM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
19/02/05 15:28:01 WARN security.UserGroupInformation: PriviledgedActionException as:TEST_USER@ADREALM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
19/02/05 15:28:01 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1549376879231
19/02/05 15:28:06 WARN security.UserGroupInformation: PriviledgedActionException as:TEST_USER@ADREALM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
19/02/05 15:28:06 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1549376879231
Related to the error above, we see this message in the file /var/log/krb5kdc.log at the master1.clouderacluster.net (machine of the local KDC):
 
Feb 05 15:31:49 master1.clouderacluster.net krb5kdc[18098](info): TGS_REQ (3 etypes {17 23 16}) 10.251.188.10: PROCESS_TGS: authtime 0,  <unknown client> for hdfs/master1.clouderacluster.net@LOCALREALM, Decrypt integrity check failed
We suspect that it might be related a the encryption mismatch, because if we compare the output of klist  -e, the difference is in the tkt ecryption section. Where AD gives 256, local ticket gives 128:
Etype (skey, tkt): aes128-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96 
 
The contents of the krb5.conf is:
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 rc4-hmac des3-cbc-sha1 des-cbc-md5
 default_tkt_enctypes = aes128-cts-hmac-sha1-96 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 rc4-hmac des3-cbc-sha1 des-cbc-md5
 default_tgs_enctypes = aes128-cts-hmac-sha1-96 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 rc4-hmac des3-cbc-sha1 des-cbc-md5
  
 default_realm = LOCALREALM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 allow-weak-crypto = true
 udp_preference_limit = 1

[realms]
 LOCALREALM = {
  kdc = master1.clouderacluster.net
  admin_server = master1.clouderacluster.net
  max_renewable_life = 7d 0h 0m 0s
  default_principal_flags = +renewable
 }
 
 ADREALM = {
  kdc = ad.domain.net:88
  admin_server = ad.domain.net:749
 }

[domain_realm]
 .localrealm = LOCALREALM
 localrealm = LOCALREALM
 .ad.domain.net = ADREALM
 ad.domain.net = ADREALM
I repeat, this happens only with the AD ticket.
All JDKs have JCE installed. 
Thanks in advance for any suggestions!
 
Who Me Too'd this topic