Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HTTP ERROR 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)

avatar
Contributor

Hey everyone, after enabling Kerberos resource manager can't run, this log after try run resource manager. please Advice

File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 295, in _run_command
    raise WebHDFSCallException(err_msg, result_dict)
resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' --negotiate -u : 'http://master.hadoop.com:50070/webhdfs/v1/services/sync/yarn-ats?op=GETFILESTATUS'' returned status_code=403. 
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)</title>
</head>
<body><h2>HTTP ERROR 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)</h2>
<table>
<tr><th>URI:</th><td>/webhdfs/v1/services/sync/yarn-ats</td></tr>
<tr><th>STATUS:</th><td>403</td></tr>
<tr><th>MESSAGE:</th><td>GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)</td></tr>
<tr><th>SERVLET:</th><td>com.sun.jersey.spi.container.servlet.ServletContainer-6f19ac19</td></tr>
</table>

</body>
</html>

  for additional informations
/etc/krb5.conf

[libdefaults]
 # renew_lifetime = 7d
  forwardable = true
  default_realm = EXAMPLE.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
#  default_tgs_enctypes = aes256-cts
 # default_tkt_enctypes = aes256-cts
  #permitted_enctypes = aes256-cts
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
  default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
  permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
[domain_realm]
  example.com = EXAMPLE.COM

[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log

[realms]
  EXAMPLE.COM = {
    master_kdc = master1.hadoop.com
    admin_server = master1.hadoop.com
    kdc = master1.hadoop.com
  }
4 REPLIES 4

avatar
Expert Contributor

You can check if the keytabs created for resource manager is equipped with AES256 encryption type or not.

Check your keytabs using below command after taking the kerberos ticket using kinit-

klist -e

avatar
Contributor

Thanks  @shubham_sharma  for the reply, I checked keytabs please see below

 

root@master:~# kinit rm/master.hadoop.com
Password for rm/master.hadoop.com@EXAMPLE.COM:
root@master:~# klist -e
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: rm/master.hadoop.com@EXAMPLE.COM

Valid starting       Expires              Service principal
07/22/2024 00:32:44  07/22/2024 10:32:44  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 07/23/2024 00:32:40, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

 

still the error, please advices

 

avatar
Expert Contributor

Hi @rizalt

There can me mismatch between your AD account and krb5.conf for encryption types[1]. Kindly check with your AD admin. 

[1] https://learn.microsoft.com/en-us/archive/blogs/openspecification/windows-configurations-for-kerbero...

avatar
Contributor

Thanks for the reply @shubham_sharma, I'm not using AD account just kerberos