Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96 while starting NN, RM & History Server UI

[#1]

Hi,

[Ambari 2.7.3, HDP 3.1]
In Active Directory Kerberized environment, I'm getting below issue when I try to access Namenode UI, RM UI and Job histroy UI from Ambari

Error:

HTTP ERROR 403
problem accessing /index.html. Reason:
GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)

krb5.conf:

  max_life = 30d
  default_tgs_enctypes = aes128-cts arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-hmac-sha1 aes256-cts
  default_tkt_enctypes = aes128-cts arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-hmac-sha1 aes256-cts
  permitted_enctypes = aes256-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-cbc-md4
  allow_weak_crypto = yes

klist:

$ls -lrt /etc/security/keytabs/spnego.service.keytab
-r--r-----. 1 root hadoop 433 Feb  9 11:59 /etc/security/keytabs/spnego.service.keytab

$klist -ket /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (arcfour-hmac)
   0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (des-cbc-md5)
   0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (aes256-cts-hmac-sha1-96)
   0 02/09/2019 07:40:04 HTTP/hostname_fqdn@realm (des3-cbc-sha1)
   0 02/09/2019 07:40:04 HTTP/hostname_fqdn@Crealm (aes128-cts-hmac-sha1-96)

kinit:

$kinit -kt /etc/security/keytabs/spnego.service.keytab $(klist -kt /etc/security/keytabs/spnego.service.keytab|sed -n "4p"|cut -d" " -f7)
# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/hostname_fqdn@realm

Valid starting       Expires              Service principal
02/09/2019 12:53:05  02/09/2019 22:53:05  krbtgt/realm@realm
        renew until 02/16/2019 12:53:05

I have re-generated the spnego keytab in all the hosts from ambari UI but did not help.

Would you please help this.

Thank you.

[#2]

@Sampath Kumar

I think there is a mismatch in the encryption types in your krb5.conf and the AD. Have a look at the below document and align your config.

Windows Configurations for Kerberos Supported Encryption Types

HTH

[#3]

Hi @Geoffrey Shelton Okot,

Thanks for the response. I have updated the krb5.conf with the below properties

# grep "enctypes" /etc/krb5.conf
 default_tgs_enctypes= des3-cbc-sha1 aes256-cts-hmac-sha1-96 arcfour-hmac aes128-cts-hmac-sha1-96 des-cbc-md5
 default_tkt_enctypes = des3-cbc-sha1 aes256-cts-hmac-sha1-96 arcfour-hmac aes128-cts-hmac-sha1-96 des-cbc-md5

# klist -aef
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/hostname_fqdn@realm
Valid starting       Expires              Service principal
02/09/2019 14:44:22  02/10/2019 00:44:22  krbtgt/realm@realm
        renew until 02/16/2019 14:44:22, Flags: FRIA
        Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
        Addresses: (none)

I don't have access to check the encryption types mapped in AD server.

Is there any way I can check this from my linux host?

Thank you.

[#4]

@Sampath Kumar

Ask your AD admin

The above should match the krb5.conf

[#5]

Hi @Geoffrey Shelton Okot,

Thanks for your time.

I have set the below two properties in core-site.xml from Ambari. Now, NN, RM and History server UI is working fine.

hadoop.http.authentication.simple.anonymous.allowed=true
hadoop.http.authentication.type=simple

Regards,

Sampath

[#6]

@Sampath Kumar

So you have disabled Kerberos for HTTP web-consoles was that intentional on a kerberized cluster or just a workaround?

[#7]

It's just a workaround @Geoffrey Shelton Okot. Thanks.

[#8]

@Sampath Kumar

If you are interested to resolve the issue then try out match the encryption types and tag me if need be?

[#9]

okay sure @Geoffrey Shelton Okot, will talk to AD team on this and let you know the status. Thanks.

[#10]

@Sampath Kumar

Cheers