Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

Highlighted

Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

Expert Contributor

Hi,

[Ambari 2.7.3, HDP 3.1]
In Active Directory Kerberized environment, I'm getting below issue when I try to access Namenode UI, RM UI and Job histroy UI from Ambari

Error:

HTTP ERROR 403
problem accessing /index.html. Reason:
GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)

krb5.conf:

  max_life = 30d
  default_tgs_enctypes = aes128-cts arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-hmac-sha1 aes256-cts
  default_tkt_enctypes = aes128-cts arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-hmac-sha1 aes256-cts
  permitted_enctypes = aes256-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-cbc-md4
  allow_weak_crypto = yes

klist:

$ls -lrt /etc/security/keytabs/spnego.service.keytab
-r--r-----. 1 root hadoop 433 Feb  9 11:59 /etc/security/keytabs/spnego.service.keytab

$klist -ket /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (arcfour-hmac)
   0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (des-cbc-md5)
   0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (aes256-cts-hmac-sha1-96)
   0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (des3-cbc-sha1)
   0 02/09/2019 07:40:04 HTTP/hostname_fqdn@Crealm (aes128-cts-hmac-sha1-96)

kinit:

$kinit -kt /etc/security/keytabs/spnego.service.keytab $(klist -kt /etc/security/keytabs/spnego.service.keytab|sed -n "4p"|cut -d" " -f7)
# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/hostname_fqdn@realm

Valid starting       Expires              Service principal
02/09/2019 12:53:05  02/09/2019 22:53:05  krbtgt/realm@realm
        renew until 02/16/2019 12:53:05

I have re-generated the spnego keytab in all the hosts from ambari UI but did not help.

Would you please help this.

Thank you.

9 REPLIES 9

Re: Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

Mentor

@Sampath Kumar

I think there is a mismatch in the encryption types in your krb5.conf and the AD. Have a look at the below document and align your config.

Windows Configurations for Kerberos Supported Encryption Types

HTH

Re: Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

Expert Contributor

Hi @Geoffrey Shelton Okot,

Thanks for the response. I have updated the krb5.conf with the below properties

# grep "enctypes" /etc/krb5.conf
 default_tgs_enctypes= des3-cbc-sha1 aes256-cts-hmac-sha1-96 arcfour-hmac aes128-cts-hmac-sha1-96 des-cbc-md5
 default_tkt_enctypes = des3-cbc-sha1 aes256-cts-hmac-sha1-96 arcfour-hmac aes128-cts-hmac-sha1-96 des-cbc-md5
# klist -aef
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/hostname_fqdn@realm
Valid starting       Expires              Service principal
02/09/2019 14:44:22  02/10/2019 00:44:22  krbtgt/realm@realm
        renew until 02/16/2019 14:44:22, Flags: FRIA
        Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
        Addresses: (none)

I don't have access to check the encryption types mapped in AD server.

Is there any way I can check this from my linux host?

Thank you.

Re: Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

Mentor

@Sampath Kumar

Ask your AD admin

103436-mit.png

The above should match the krb5.conf

Re: Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

Expert Contributor

Hi @Geoffrey Shelton Okot,

Thanks for your time.

I have set the below two properties in core-site.xml from Ambari. Now, NN, RM and History server UI is working fine.

hadoop.http.authentication.simple.anonymous.allowed=true
hadoop.http.authentication.type=simple

Regards,

Sampath

Re: Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

Mentor

@Sampath Kumar

So you have disabled Kerberos for HTTP web-consoles was that intentional on a kerberized cluster or just a workaround?

Re: Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

Expert Contributor

It's just a workaround @Geoffrey Shelton Okot. Thanks.

Re: Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

Mentor

@Sampath Kumar

If you are interested to resolve the issue then try out match the encryption types and tag me if need be?

Re: Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

Expert Contributor

okay sure @Geoffrey Shelton Okot, will talk to AD team on this and let you know the status. Thanks.

Re: Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

Mentor
Don't have an account?
Coming from Hortonworks? Activate your account here