Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Kerberos not authenticating from Hadoop Gateway node

avatar

All,

I have a HDFS/Spark/YARN gateway node that has been assigned the roles via cloudera manager. However, Kerberos is preventing me from accessing HDFS from the gateway node. I could generate TGT via kinit command. I could access HDFS from other nodes in the hadoop cluster using the same user id and same steps.

All kerberos configuration files in the gateway node seems to be in place. Any suggestions what i could have missed?

 

 

2 ACCEPTED SOLUTIONS

avatar
Master Guru

Hi @VamshiDevraj ,

 

Since your klist shows you do have a TGT but Java cannot find it, the most likely cause is that the version of Java you are using does not support aes256-cts-hmac-sha1-96 encryption as it is configured.  Can you try doing "java -version" to find out what version of Java your client is using?

 

Note the changes that have occurred as mentioned on this page:

https://golb.hplar.ch/2017/10/JCE-policy-changes-in-Java-SE-8u151-and-8u152.html

 

If your client is using a version of the JDK older than 1.8 update 162, some sort of configuration needs to be done in that JDK to allow it to support AES256 and be able to access your TGT.

 

Given what you showed us, there is a relatively high probability this will help.

View solution in original post

avatar
Expert Contributor

@VamshiDevraj  If you are still facing issue can you share details about the error or screenshot for the same?

View solution in original post

5 REPLIES 5

avatar
Guru

Hi @VamshiDevraj ,

 

Could you please share the steps and error messages you are getting? 

 

Thanks,

Li

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

avatar

 

[centos@cdws user]$ hostname -f
cdws.company.fr

[centos@cdws user]$ kinit
Password for centos@COMPANY.COM:
[centos@cdws user]$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: centos@COMPANY.COM

Valid starting Expires Service principal
11/12/2019 21:13:35 11/13/2019 21:13:35 krbtgt/COMPANY.COM@COMPANY.COM

Below is the HDFS command

[centos@cdws user]$ hdfs dfs -ls /user/centos
19/11/12 22:29:28 WARN security.UserGroupInformation: PriviledgedActionException as:centos (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
19/11/12 22:29:28 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
19/11/12 22:29:28 WARN security.UserGroupInformation: PriviledgedActionException as:centos (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "cdws.company.fr/172.16.40.7"; destination host is: "master.company.fr":8020;

 

[centos@cdws user]$ sudo yum list installed | grep krb5
krb5-devel.x86_64 1.15.1-37.el7_7.2 @updates
krb5-libs.x86_64 1.15.1-37.el7_7.2 @updates
krb5-workstation.x86_64 1.15.1-37.el7_7.2 @updates

 

[centos@cdws user]$ sudo cat /etc/krb5.conf
[libdefaults]
default_realm = COMPANY.COM
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts-hmac-sha1-96
default_tkt_enctypes = aes256-cts-hmac-sha1-96
permitted_enctypes = aes256-cts-hmac-sha1-96
udp_preference_limit = 1
kdc_timeout = 3000
[realms]
COMPANY.COM = {
kdc = worker3.company.fr
admin_server = worker3.company.fr
}
[domain_realm]

 

------------------------------------------------------------------------

 

However, one thing worth noting is i don't see any principle for "cdws.company.fr" in kerberos credential tab in CM. Is this causing the issue and how could I resolve it?

 

Many thanks,

 

avatar
Master Guru

Hi @VamshiDevraj ,

 

Since your klist shows you do have a TGT but Java cannot find it, the most likely cause is that the version of Java you are using does not support aes256-cts-hmac-sha1-96 encryption as it is configured.  Can you try doing "java -version" to find out what version of Java your client is using?

 

Note the changes that have occurred as mentioned on this page:

https://golb.hplar.ch/2017/10/JCE-policy-changes-in-Java-SE-8u151-and-8u152.html

 

If your client is using a version of the JDK older than 1.8 update 162, some sort of configuration needs to be done in that JDK to allow it to support AES256 and be able to access your TGT.

 

Given what you showed us, there is a relatively high probability this will help.

avatar

Hi @bgooley ,

 

I see your point. I realized now that the JDK on the gateway node is 1.6 while all other nodes are on JDK1.8. This indeed could be causing the problem.

 

I'll first upgrade to jdk1.8 and attempt later....thank you so much.

 

Regards,

Vamshi

avatar
Expert Contributor

@VamshiDevraj  If you are still facing issue can you share details about the error or screenshot for the same?