Kerberos not authenticating from Hadoop Gateway node

[#1]

All,

I have a HDFS/Spark/YARN gateway node that has been assigned the roles via cloudera manager. However, Kerberos is preventing me from accessing HDFS from the gateway node. I could generate TGT via kinit command. I could access HDFS from other nodes in the hadoop cluster using the same user id and same steps.

All kerberos configuration files in the gateway node seems to be in place. Any suggestions what i could have missed?

[#2 · Accepted solution]

Hi @VamshiDevraj ,

Since your klist shows you do have a TGT but Java cannot find it, the most likely cause is that the version of Java you are using does not support aes256-cts-hmac-sha1-96 encryption as it is configured.  Can you try doing "java -version" to find out what version of Java your client is using?

Note the changes that have occurred as mentioned on this page:

https://golb.hplar.ch/2017/10/JCE-policy-changes-in-Java-SE-8u151-and-8u152.html

If your client is using a version of the JDK older than 1.8 update 162, some sort of configuration needs to be done in that JDK to allow it to support AES256 and be able to access your TGT.

Given what you showed us, there is a relatively high probability this will help.

[#3]

Hi @VamshiDevraj ,

Could you please share the steps and error messages you are getting? 

Thanks,

Li

[#4]

[centos@cdws user]$ hostname -f
cdws.company.fr

[centos@cdws user]$ kinit
Password for centos@COMPANY.COM:
[centos@cdws user]$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: centos@COMPANY.COM

Valid starting Expires Service principal
11/12/2019 21:13:35 11/13/2019 21:13:35 krbtgt/COMPANY.COM@COMPANY.COM

Below is the HDFS command

[centos@cdws user]$ hdfs dfs -ls /user/centos
19/11/12 22:29:28 WARN security.UserGroupInformation: PriviledgedActionException as:centos (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
19/11/12 22:29:28 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
19/11/12 22:29:28 WARN security.UserGroupInformation: PriviledgedActionException as:centos (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "cdws.company.fr/172.16.40.7"; destination host is: "master.company.fr":8020;

[centos@cdws user]$ sudo yum list installed | grep krb5
krb5-devel.x86_64 1.15.1-37.el7_7.2 @updates
krb5-libs.x86_64 1.15.1-37.el7_7.2 @updates
krb5-workstation.x86_64 1.15.1-37.el7_7.2 @updates

[centos@cdws user]$ sudo cat /etc/krb5.conf
[libdefaults]
default_realm = COMPANY.COM
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts-hmac-sha1-96
default_tkt_enctypes = aes256-cts-hmac-sha1-96
permitted_enctypes = aes256-cts-hmac-sha1-96
udp_preference_limit = 1
kdc_timeout = 3000
[realms]
COMPANY.COM = {
kdc = worker3.company.fr
admin_server = worker3.company.fr
}
[domain_realm]

------------------------------------------------------------------------

However, one thing worth noting is i don't see any principle for "cdws.company.fr" in kerberos credential tab in CM. Is this causing the issue and how could I resolve it?

Many thanks,

[#5 · Accepted solution]

Hi @VamshiDevraj ,

Since your klist shows you do have a TGT but Java cannot find it, the most likely cause is that the version of Java you are using does not support aes256-cts-hmac-sha1-96 encryption as it is configured.  Can you try doing "java -version" to find out what version of Java your client is using?

Note the changes that have occurred as mentioned on this page:

https://golb.hplar.ch/2017/10/JCE-policy-changes-in-Java-SE-8u151-and-8u152.html

If your client is using a version of the JDK older than 1.8 update 162, some sort of configuration needs to be done in that JDK to allow it to support AES256 and be able to access your TGT.

Given what you showed us, there is a relatively high probability this will help.

[#6]

Hi @bgooley ,

I see your point. I realized now that the JDK on the gateway node is 1.6 while all other nodes are on JDK1.8. This indeed could be causing the problem.

I'll first upgrade to jdk1.8 and attempt later....thank you so much.

Regards,

Vamshi

[#7 · Accepted solution]

@VamshiDevraj  If you are still facing issue can you share details about the error or screenshot for the same?