Kerberos setup failed on cluster due zookeeper error

[#1]

Hello,

We are running CDH 5.14.0.We were setting up kerberos on our cluster.For this we were trying with MIT kerberos.Please see the Krb5.conf and kdc.conf files details as below.

krb5.conf:

]$ cat /etc/krb5.conf
[libdefaults]
default_realm = HADOOP.COM
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 arcfour-hmac-md5
default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 arcfour-hmac-md5
permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 arcfour-hmac-md5

[realms]
HADOOP.COM = {
kdc = hadoop-admin
admin_server = hadoop-admin
}

kdc.conf:

sudo cat /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88

[realms]
HADOOP.COM = {
#master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal des3-hmac-sha1:normal arcfour-hmac-md5:normal
max_renewable_life = 7d
}

During setup it failed with zookeeper error.We checked the error logs for zookeeper.Please find error :

[root@hadoop-slave-1 ~]# tail -f /var/log/zookeeper/zookeeper-cmf-zookeeper-SERVER-hadoop-slave-1.log
2020-03-06 17:08:16,659 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task started.
2020-03-06 17:08:16,672 INFO org.apache.zookeeper.server.quorum.QuorumPeerMain: Starting quorum peer
2020-03-06 17:08:16,676 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task completed.
2020-03-06 17:08:16,768 ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unexpected exception, exiting abnormally
java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth. login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)

Do anybody have an idea on that.Please share if you have solution.

Thanks

pdev

[#2]

This can mean one of a few things but ultimately the error code you are seeing is being returned by your KDC.

Client not found in database means that the requested SPN cannot be found by the KDC. This most often occurs when  both forward and reverse DNS are not properly configured in your environment. If you find the both forward and reverse DNS are in working order then you should review credentials on the KDC and ensure that the credentials we are trying to use exists for this host.

You can enable Kerberos debugging to get additional information from the JVM on the Kerberos interaction but generally speaking zookeeper uses one of two credentials. Either

HTTP/<fqdn>@realm

or

zookeeper/<fqdn>@realm.

[#3]

Hi,

Thanks for the reply.Actually  we do not have any DNS.But when this error came,then for cross verification i Checked all principles in KDC server.

In those principles , zookeeper service principal was absent. Dont know the reason,whycloudera manager was not able to create principal only for zookeeper.

Waiting for your response.

Thanks 

Pdev