Kerberos - Issue

[#1]

Can anyone explain whats the issue based on the error log from Namenode .

Also because of this issue , Namenode is in Safe Mode. 

I had manually configured Kerberos and made the Cluster kerberoised using Cloudera Manager 

Its really driving me insane . I am really tired of configuring kerberos in QuickStart 

Socket Reader #1 for port 8020: readAndProcess from client 192.168.19.131 threw exception [javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]]

[#2]

I have the policy jar in the right directory 

[cloudera@quickstart jdk1.7.0_67-cloudera]$ cd jre/lib/security/
[cloudera@quickstart security]$ ls
blacklist      java.policy    local_policy.jar
cacerts        java.security  trusted.libraries
javafx.policy  javaws.policy  US_export_policy.jar

This the debug trace 

[cloudera@quickstart jdk1.7.0_67-cloudera]$ export HADOOP_OPTS="-Dsun.security.krb5.debug=true"
[cloudera@quickstart jdk1.7.0_67-cloudera]$ hadoop fs -ls 
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
>>>KinitOptions cache name is /tmp/krb5cc_501
>>>DEBUG <CCacheInputStream>  client principal is hdfs@HADOOPSEC.COM
>>>DEBUG <CCacheInputStream> server principal is krbtgt/HADOOPSEC.COM@HADOOPSEC.COM
>>>DEBUG <CCacheInputStream> key type: 18
>>>DEBUG <CCacheInputStream> auth time: Mon May 14 10:25:41 PDT 2018
>>>DEBUG <CCacheInputStream> start time: Mon May 14 10:25:41 PDT 2018
>>>DEBUG <CCacheInputStream> end time: Tue May 15 10:25:41 PDT 2018
>>>DEBUG <CCacheInputStream> renew_till time: Mon May 21 10:25:41 PDT 2018
>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE; INITIAL;
>>>DEBUG <CCacheInputStream>  client principal is hdfs@HADOOPSEC.COM
>>>DEBUG <CCacheInputStream> server principal is X-CACHECONF:/krb5_ccache_conf_data/fast_avail/krbtgt/HADOOPSEC.COM@HADOOPSEC.COM
>>>DEBUG <CCacheInputStream> key type: 0
>>>DEBUG <CCacheInputStream> auth time: Wed Dec 31 16:00:00 PST 1969
>>>DEBUG <CCacheInputStream> start time: null
>>>DEBUG <CCacheInputStream> end time: Wed Dec 31 16:00:00 PST 1969
>>>DEBUG <CCacheInputStream> renew_till time: null
>>> CCacheInputStream: readFlags() 
>>> unsupported key type found the default TGT: 18
18/05/14 10:27:37 WARN security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
18/05/14 10:27:37 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
18/05/14 10:27:37 WARN security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "quickstart.cloudera/192.168.19.131"; destination host is: "quickstart.cloudera":8020;

 My klist 

[cloudera@quickstart jdk1.7.0_67-cloudera]$ klist -e
Ticket cache: FILE:/tmp/krb5cc_501
Default principal: hdfs@HADOOPSEC.COM

Valid starting     Expires            Service principal
05/14/18 10:25:41  05/15/18 10:25:41  krbtgt/HADOOPSEC.COM@HADOOPSEC.COM
	renew until 05/21/18 10:25:41, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 

[#3]

@cjervis Could you please help me out with this issue . Thanks 

[#4]

Hey @MattSun,

I'm not your best source for this as my expertise is in Community Management but sometimes I get lucky so I read over your post. I zeroed in on this part of the initial issue.

Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled

So I did a search on the community to see what would pop up and found this thread. I also saw a mention of JCE policy so I'll provide a link to documention on that as well.

Give them both a read to see if they apply to your situation. 

[#5]

Thanks for the quick turn around. 

As pointed in my previous thread , i had showed the snap shot that I had my policy jar in place .

But it is still erroring out more over Cloudera Quickstart Vm  do comes with Policy jars inside jre/lib/security . 

i would really appreciate if anyone can help me , Because it is quickstart vm thought  someone would pitch in . 

@cjervis

Matt 

[#6]

Hi @MattSun

Are those the unlimited strength policy JARs?

Run this:

ls -l /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/*policy.jar

The file sizes should be 2500 for local_policy.jar and 2487 for US_export_policy.jar.

If they're 2865 and 2397, then those are the ones that are included with the JDK, and are not the unlimited strength ones you need to enable AES256.

[#7]

@Brian Burton

I performed ls on the java folder , the number is aint matching . i am sorry could you please take a look of the output see if that is unlimited strength ones that i need to enable AES256  . 

[#8]

@Brian Burton

Below is the One that i downloaded from Oracle website looks like this . So should i remove the previous policy jar and replace with the below . ? Could you let me know . 

-rw-r--r--@ 1 matt staff  7289 May 31  2011 README.txt
-rw-rw-r--@ 1 matt   staff  2487 May 31  2011 US_export_policy.jar
-rw-rw-r--@ 1 matt   staff  2500 May 31  2011 local_policy.jar

[#9 · Accepted solution]

@MattSun

Yes, remove the old ones and replace them with those that you downloaded from Oracle.

[#10]

Also, if you'd like to check if the JCE files you have are of unlimited policy type, you can follow this: http://harshj.com/checking-if-your-jre-has-the-unlimited-strength-policy-files-in-place/

Also, a note: In latest JDK8 update and in all JDK9+, the unlimited cryptography policies are shipped and active by default. This step of manually replacing the JCE jars is only required for JDK7 and early JDK8 releases. The QuickStart VM uses JDK7 currently.