Not able to connect to OOZIE server after kerberizing "Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96"

[#1]

Hello, We are setting up a staging cluster with kerberos enabled. After enabling kerberos on oozie, we are able to start oozie server but not able to connect to the server. In oozie error log we see following error

Authentication exception: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) -Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96) org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)

Keytabs are generated via ambari and following are the details of the same.

user1@oozie1001:~$ sudo -u oozie klist -kte /etc/security/keytabs/oozie.service.keytab
Keytab name: FILE:/etc/security/keytabs/oozie.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
1 07/24/2017 10:53:19 oozie/oozie1001.example.com@EXAMPLE.COM (des3-cbc-sha1)
1 07/24/2017 10:53:19 oozie/oozie1001.example.com@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
1 07/24/2017 10:53:19 oozie/oozie1001.example.com@EXAMPLE.COM (arcfour-hmac)
1 07/24/2017 10:53:19 oozie/oozie1001.example.com@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
1 07/24/2017 10:53:19 oozie/oozie1001.example.com@EXAMPLE.COM (des-cbc-md5)

user1@oozie1001:~$ sudo -u oozie klist -e
Ticket cache: FILE:/tmp/krb5cc_1183
Default principal: oozie/oozie1001.example.com@EXAMPLE.COM
Valid starting ExpiresService principal07/25/2017 08:27:4507/25/2017 18:27:45krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 08/01/2017 08:27:45, Etype (skey, tkt): des3-cbc-sha1, aes256-cts-hmac-sha1-96

user1@oozie1001:~ cat /etc/krb5.conf |grep enctypes
default_tgs_enctypes= des3-cbc-sha1 aes256-cts-hmac-sha1-96 arcfour-hmac aes128-cts-hmac-sha1-96 des-cbc-md5
default_tkt_enctypes = des3-cbc-sha1 aes256-cts-hmac-sha1-96 arcfour-hmac aes128-cts-hmac-sha1-96 des-cbc-md5

While starting the oozie server it is able to get tgt for oozie principal

user1@oozie1001:~$ grep keytab /var/log/oozie/oozie.log
2017-07-25 09:12:18,161INFO main HadoopAccessorService - SERVER[oozie1001.example.com] Got Kerberos ticket, keytab [/etc/security/keytabs/oozie.service.keytab], Oozie principal principal [oozie/oozie1001.example.com@EXAMPLE.COM]

To test the oozie we try following command and it ends up in Authentication failed.

user1@oozie1001:~$ sudo -u oozie /opt/gridops/oozie/oozie_current/bin/oozie admin -status
Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, status: 403, message: Forbidden

On KDC we are able to see that authentication is successful for oozie/oozie1001.example.com@EXAMPLE.COM

Jul 25 11:50:50 kdcserver1000 krb5kdc[959]: TGS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: ISSUE: authtime 1500971265, etypes {rep=16 tkt=18 ses=16}, oozie/oozie1001.example.com@EXAMPLE.COM for HTTP/oozie1001.example.com@EXAMPLE.COM
Jul 25 11:50:50 kdcserver1000 krb5kdc[959]: TGS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: ISSUE: authtime 1500971265, etypes {rep=16 tkt=18 ses=16}, oozie/oozie1001.example.com@EXAMPLE.COM for HTTP/oozie1001.example.com@EXAMPLE.COM
Jul 25 11:51:18 kdcserver1000 krb5kdc[959]: AS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: NEEDED_PREAUTH: oozie/oozie1001.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required
Jul 25 11:51:18 kdcserver1000 krb5kdc[959]: AS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: ISSUE: authtime 1500983478, etypes {rep=16 tkt=18 ses=16}, oozie/oozie1001.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Jul 25 11:51:20 kdcserver1000 krb5kdc[959]: AS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: NEEDED_PREAUTH: oozie/oozie1001.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required
Jul 25 11:51:20 kdcserver1000 krb5kdc[959]: AS_REQ (4 etypes {16 18 23 17}) 192.168.142.13: ISSUE: authtime 1500983480, etypes {rep=16 tkt=18 ses=16}, oozie/oozie1001.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM

Oozie log file:

2017-07-25 09:09:20,173WARN http-11000-2 AuthenticationFilter - SERVER[oozie1001.example.com] Authentication exception: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)

JCE is configured on the server.

user1@oozie1001:~$ zipgrep CryptoAllPermission $JAVA_HOME/jre/lib/security/local_policy.jar
default_local.policy:permission javax.crypto.CryptoAllPermission;

user1@oozie1001:~$ cat Test-JCE.java
import javax.crypto.Cipher;
class Test {
  public static void main(String[] args) {
    try {
       int maxKeyLen = Cipher.getMaxAllowedKeyLength("AES");
       System.out.println(maxKeyLen);
    } catch (Exception e)
    {System.out.println("JCE not enable :(");}}}
user1@oozie1001:~$ java Test
2147483647

spnego file is set to 640

user1@oozie001:~$ ls -al /etc/security/keytabs/spnego.service.keytab
-rw-r----- 1 root hadoop 438 Jul 24 09:49 /etc/security/keytabs/spnego.service.keytab

nslookup oozie1001.example.com is resolving.

Kerberos version is same in both KDC as well as the keytab

getprinc oozie/oozie1001.example.com@EXAMPLE.COM
Principal: oozie/oozie1001.example.com@EXAMPLE.COM
Expiration date: [never]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 8
Key: vno 1, aes256-cts-hmac-sha1-96, no salt
Key: vno 1, arcfour-hmac, no salt
Key: vno 1, des3-cbc-sha1, no salt
Key: vno 1, des-cbc-crc, no salt
Key: vno 1, des-cbc-md5, Version 4
Key: vno 1, des-cbc-md5, Version 5 - No Realm
Key: vno 1, des-cbc-md5, Version 5 - Realm Only
Key: vno 1, des-cbc-md5, AFS version 3
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH

Any help in what else did I miss ?

Thanks in advance,

[#2 · Accepted solution]

Hello @Mazin Mohammed,

Thanks for taking time to write a detailed question. It looks like that the problem is not with Oozie keytab but spnego (HTTP) keytab. Please see if you can run this and share output here:

kinit <end-user>
kvno HTTP/oozie1001.example.com@EXAMPLE.COM
klist -eaf

klist -kte /etc/security/keytabs/spnego.service.keytab

getprinc HTTP/oozie1001.example.com@EXAMPLE.COM

Regards.

[#3]

mazin.mohammed@oozie1001:~$ klist -eaf
Ticket cache: FILE:/tmp/krb5cc_1993
Default principal: mazin.mohammed@EXAMPLE.COM
Valid starting ExpiresService principal
07/31/2017 04:03:3507/31/2017 14:03:35krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 08/07/2017 04:03:31, Flags: FRIA
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
Addresses: (none)

mazin.mohammed@oozie1001:~$ ls -al /etc/security/keytabs/spnego.service.keytab
-rw-r----- 1 root hadoop 438 Jul 24 04:50 /etc/security/keytabs/spnego.service.keytab

mazin.mohammed@oozie1001:~$ sudo klist -kte /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
1 07/24/2017 04:50:27 HTTP/oozie1001.example.com@EXAMPLE.COM (des3-cbc-sha1)
1 07/24/2017 04:50:27 HTTP/oozie1001.example.com@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
1 07/24/2017 04:50:27 HTTP/oozie1001.example.com@EXAMPLE.COM (arcfour-hmac)
1 07/24/2017 04:50:27 HTTP/oozie1001.example.com@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
1 07/24/2017 04:50:27 HTTP/oozie1001.example.com@EXAMPLE.COM (des-cbc-md5)

kadmin: getprinc HTTP/oozie1001.example.com@EXAMPLE.COM

Principal: HTTP/oozie1001.example.com@EXAMPLE.COM
Expiration date: [never]
Last password change: Mon Jul 24 04:50:25 UTC 2017
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Key: vno 1, aes256-cts-hmac-sha1-96, no salt
Key: vno 1, arcfour-hmac, no salt
Key: vno 1, des3-cbc-sha1, no salt
Key: vno 1, des-cbc-crc, no salt
Key: vno 1, des-cbc-md5, Version 4
Key: vno 1, des-cbc-md5, Version 5 - No Realm
Key: vno 1, des-cbc-md5, Version 5 - Realm Only
Key: vno 1, des-cbc-md5, AFS version 3
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: [none]

[#4]

@Vipin Rathor

Thanks for the pointer. The issue was with spnego.service.keytab. But it was was permission issue. Once we got the user oozie to be part of hadoop group the issue was fixed. We do run the daemon as oozie user and thus it was not able to read the keytab file.

Thanks

[#5]

@Mazin Mohammed

Great to hear that your problem has been resolved. If you find my reply useful, please accept it as answer, so that other facing similar issue can know what to look for. Thanks.