Client not found in kerberos database error

[#1]

Hello,

All services are failing post enabling kerberos with error - "client not found in kerberos database"

Kinit yields the same error while using svchdfs account through keytab. kinit to svchdfs works fine if logged in through password. Same error post regenerating keytabs.

Appreciate any pointers.

1) HDP 2.3.4.0, Ambari 2.2.0.

2) Pre-created service account are used.

3) AD as Kerberos.

4) AD Structure

OU ---level1---> HADOOP

---level1---> cluster1 - serviceprincipals

---level1---> PROD

--------level2--------> cluster2 serviceprincipals

cluster1 is working fine, cluster2 fails.

Regards

PranayVyas

[#2]

Hi Jason,

1) Klist from svchdfs says not ticket cache

2) Klist of keytab shows svchdfs-<clustername>@REALM.COM

3) kinit -kt hdfs.headless.keytab svchdfs-<clustername>

We noticed that svchdfs-<clustername> exists at 2 OU's within AD. That could be a cause since kerberos is unable to uniquely identify service account. we are trying to delete the duplicate one.

Regards

Pranay Vyas

[#3]

Check if the Kerberos realm name in AD is in lowercase. I have seen this problem if that is the case. If it is, you would be able to complete the Kerberos wizard, but service startup will fail with this error. The MIT KDC libraries require the realm to be uppercase for things to work properly.

[#4 · Accepted solution]

Thanks emaxwell and Jason. The problem was due to duplicate HTTP and http account in AD. Deleting the centirfy's 'http' account resolved all issues.

[#5]

I accepted your answer as we want to show exact solution, which was different from what was suggested by others.

[#6]

As we have been bitten by the AD issues mentioned by @Pranay Vyas. I thought I'd expand upon the issue.

We wanted two clusters as similar as possible for DR purposes and was looking at using different AD OU's but the same cluster name. Please note as in HDP 2.5.5 Ambari 2.4.2, keytabs will be generated following the "name-cluster-name" pattern (i.e. ambari-qa-sandpit).

You can create the two sets of AD principals but it fails (usually around Zookeeper) with the issue "client not found in kerberos database" even though you can see the entities in AD or via an ldapsearch. This means by default you can't have two clusters with the same name connected to the same AD.

We didn't investigate changing the kerberos naming pattern but this could possibly fix the issue.