Created 07-17-2024 09:17 PM
Hey everyone, after enabling Kerberos resource manager can't run, this log after try run resource manager. please Advice
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 295, in _run_command
raise WebHDFSCallException(err_msg, result_dict)
resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' --negotiate -u : 'http://master.hadoop.com:50070/webhdfs/v1/services/sync/yarn-ats?op=GETFILESTATUS'' returned status_code=403.
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)</title>
</head>
<body><h2>HTTP ERROR 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)</h2>
<table>
<tr><th>URI:</th><td>/webhdfs/v1/services/sync/yarn-ats</td></tr>
<tr><th>STATUS:</th><td>403</td></tr>
<tr><th>MESSAGE:</th><td>GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)</td></tr>
<tr><th>SERVLET:</th><td>com.sun.jersey.spi.container.servlet.ServletContainer-6f19ac19</td></tr>
</table>
</body>
</html>
for additional informations
/etc/krb5.conf
[libdefaults]
# renew_lifetime = 7d
forwardable = true
default_realm = EXAMPLE.COM
ticket_lifetime = 24h
dns_lookup_realm = false
dns_lookup_kdc = false
default_ccache_name = /tmp/krb5cc_%{uid}
# default_tgs_enctypes = aes256-cts
# default_tkt_enctypes = aes256-cts
#permitted_enctypes = aes256-cts
#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
[domain_realm]
example.com = EXAMPLE.COM
[logging]
default = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
kdc = FILE:/var/log/krb5kdc.log
[realms]
EXAMPLE.COM = {
master_kdc = master1.hadoop.com
admin_server = master1.hadoop.com
kdc = master1.hadoop.com
}
Created on 07-19-2024 07:32 AM - edited 07-19-2024 07:35 AM
You can check if the keytabs created for resource manager is equipped with AES256 encryption type or not.
Check your keytabs using below command after taking the kerberos ticket using kinit-
klist -e
Created on 07-21-2024 05:39 PM - edited 07-21-2024 05:40 PM
Thanks @shubham_sharma for the reply, I checked keytabs please see below
root@master:~# kinit rm/master.hadoop.com
Password for rm/master.hadoop.com@EXAMPLE.COM:
root@master:~# klist -e
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: rm/master.hadoop.com@EXAMPLE.COM
Valid starting Expires Service principal
07/22/2024 00:32:44 07/22/2024 10:32:44 krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 07/23/2024 00:32:40, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
still the error, please advices
Created 07-22-2024 12:46 PM
Hi @rizalt
There can me mismatch between your AD account and krb5.conf for encryption types[1]. Kindly check with your AD admin.
Created 07-22-2024 06:03 PM
Thanks for the reply @shubham_sharma, I'm not using AD account just kerberos