Hello all,
after fresh kerberization of Ambari 2.7.3 / HDP 3 cluster, the HDFS namenode isn't able to start because the hdfs user can't talk to the webhdfs. The following error is returned:
GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
It is not only from ambari: I can recreate this error from a simple curl call from hdfs user:
su - hdfs
curl --negotiate -u : http://datanode:50070/webhdfs/v1/tmp?op=GETFILESTATUS
Which returns
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /webhdfs/v1/tmp. Reason:
<pre> GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)</pre></p>
</body>
</html>
Overall permission for this user should be in tact, since I'm able to run hdfs operations from shell and kinit without problems. What could be the problem?
I've tried recreating keytabs several times, and fiddling with ACL settings on the config, but nothing works. What principal is WEBHDFS expecting? The same results are when I'm trying accessing it with HTTP/host@EXAMPLE.COM principal.
NB: I'll add that there's nothing fancy in the HDFS settings, mainly stock/default config.
NB2: I will add, that I've added all possible encryption types to krb5.conf as I could find, but none if these helped:
default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
permitted_enctypes = aes256-cts-hmac-sha1-96 aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
@Geoffrey Shelton Okot
