Hello,
We are setting up a staging cluster with kerberos enabled. After enabling kerberos on oozie, we are able to start oozie server but not able to connect to the server. In oozie error log we see following error Authentication exception: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) -Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96) org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96) Keytabs are generated via ambari and following are the details of the same. user1@oozie1001:~$ sudo -u oozie klist -kte /etc/security/keytabs/oozie.service.keytab
Keytab name: FILE:/etc/security/keytabs/oozie.service.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
1 07/24/2017 10:53:19 oozie/oozie1001.example.com@EXAMPLE.COM (des3-cbc-sha1)
1 07/24/2017 10:53:19 oozie/oozie1001.example.com@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
1 07/24/2017 10:53:19 oozie/oozie1001.example.com@EXAMPLE.COM (arcfour-hmac)
1 07/24/2017 10:53:19 oozie/oozie1001.example.com@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
1 07/24/2017 10:53:19 oozie/oozie1001.example.com@EXAMPLE.COM (des-cbc-md5) user1@oozie1001:~$ sudo -u oozie klist -e
Ticket cache: FILE:/tmp/krb5cc_1183
Default principal: oozie/oozie1001.example.com@EXAMPLE.COM
Valid starting ExpiresService principal07/25/2017 08:27:4507/25/2017 18:27:45krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 08/01/2017 08:27:45, Etype (skey, tkt): des3-cbc-sha1, aes256-cts-hmac-sha1-96 user1@oozie1001:~ cat /etc/krb5.conf |grep enctypes
default_tgs_enctypes= des3-cbc-sha1 aes256-cts-hmac-sha1-96 arcfour-hmac aes128-cts-hmac-sha1-96 des-cbc-md5
default_tkt_enctypes = des3-cbc-sha1 aes256-cts-hmac-sha1-96 arcfour-hmac aes128-cts-hmac-sha1-96 des-cbc-md5 While starting the oozie server it is able to get tgt for oozie principal user1@oozie1001:~$ grep keytab /var/log/oozie/oozie.log
2017-07-25 09:12:18,161INFO main HadoopAccessorService - SERVER[oozie1001.example.com] Got Kerberos ticket, keytab [/etc/security/keytabs/oozie.service.keytab], Oozie principal principal [oozie/oozie1001.example.com@EXAMPLE.COM] To test the oozie we try following command and it ends up in Authentication failed. user1@oozie1001:~$ sudo -u oozie /opt/gridops/oozie/oozie_current/bin/oozie admin -status
Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, status: 403, message: Forbidden On KDC we are able to see that authentication is successful for oozie/oozie1001.example.com@EXAMPLE.COM Jul 25 11:50:50 kdcserver1000 krb5kdc[959]: TGS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: ISSUE: authtime 1500971265, etypes {rep=16 tkt=18 ses=16}, oozie/oozie1001.example.com@EXAMPLE.COM for HTTP/oozie1001.example.com@EXAMPLE.COM
Jul 25 11:50:50 kdcserver1000 krb5kdc[959]: TGS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: ISSUE: authtime 1500971265, etypes {rep=16 tkt=18 ses=16}, oozie/oozie1001.example.com@EXAMPLE.COM for HTTP/oozie1001.example.com@EXAMPLE.COM
Jul 25 11:51:18 kdcserver1000 krb5kdc[959]: AS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: NEEDED_PREAUTH: oozie/oozie1001.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required
Jul 25 11:51:18 kdcserver1000 krb5kdc[959]: AS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: ISSUE: authtime 1500983478, etypes {rep=16 tkt=18 ses=16}, oozie/oozie1001.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Jul 25 11:51:20 kdcserver1000 krb5kdc[959]: AS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: NEEDED_PREAUTH: oozie/oozie1001.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required
Jul 25 11:51:20 kdcserver1000 krb5kdc[959]: AS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: ISSUE: authtime 1500983480, etypes {rep=16 tkt=18 ses=16}, oozie/oozie1001.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM Oozie log file: 2017-07-25 09:09:20,173WARN http-11000-2 AuthenticationFilter - SERVER[oozie1001.example.com] Authentication exception: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399) JCE is configured on the server. user1@oozie1001:~$ zipgrep CryptoAllPermission $JAVA_HOME/jre/lib/security/local_policy.jar
default_local.policy:permission javax.crypto.CryptoAllPermission; user1@oozie1001:~$ cat Test-JCE.java
import javax.crypto.Cipher;
class Test {
public static void main(String[] args) {
try {
int maxKeyLen = Cipher.getMaxAllowedKeyLength("AES");
System.out.println(maxKeyLen);
} catch (Exception e)
{System.out.println("JCE not enable :(");}}}
user1@oozie1001:~$ java Test
2147483647 spnego file is set to 640 user1@oozie001:~$ ls -al /etc/security/keytabs/spnego.service.keytab
-rw-r----- 1 root hadoop 438 Jul 24 09:49 /etc/security/keytabs/spnego.service.keytab nslookup oozie1001.example.com is resolving. Kerberos version is same in both KDC as well as the keytab getprinc oozie/oozie1001.example.com@EXAMPLE.COM
Principal: oozie/oozie1001.example.com@EXAMPLE.COM
Expiration date: [never]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 8
Key: vno 1, aes256-cts-hmac-sha1-96, no salt
Key: vno 1, arcfour-hmac, no salt
Key: vno 1, des3-cbc-sha1, no salt
Key: vno 1, des-cbc-crc, no salt
Key: vno 1, des-cbc-md5, Version 4
Key: vno 1, des-cbc-md5, Version 5 - No Realm
Key: vno 1, des-cbc-md5, Version 5 - Realm Only
Key: vno 1, des-cbc-md5, AFS version 3
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH Any help in what else did I miss ? Thanks in advance,
... View more
