Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Hive Service won't start. LoginException: Client not found in Kerberos database

Labels:
avatar
author-rank kal
Explorer

Created on ‎08-30-2019 07:20 AM - last edited on ‎09-01-2019 05:09 AM by Community Manager Community Manager

Hive Service won't start (HiveMetaStore [main]: org.apache.thrift.transport.TTransportException: java.io.IOException: Login failure for hive/xxxx.sys.xxxx.net@REALM.NET from keytab hive.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database )

HiveMetaStore 
[main]: org.apache.thrift.transport.TTransportException: java.io.IOException: Login failure for hive/shive/xxxx.sys.xxxx.net@REALM.NET from keytab hive.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.<init>(HadoopThriftAuthBridge.java:358)
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createServer(HadoopThriftAuthBridge.java:102)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6138)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6057)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Login failure for hive/spectra-as-z15p.sys.comcast.net@SPECTRA.COMCAST.NET from keytab hive.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962)
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.<init>(HadoopThriftAuthBridge.java:353)
	... 9 more
Caused by: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:953)
	... 10 more
Caused by: KrbException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:82)
	at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
	at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
	... 23 more
Caused by: KrbException: Identifier doesn't match expected value (906)
	at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
	at sun.security.krb5.internal.ASRep.init(ASRep.java:64)
	at sun.security.krb5.internal.ASRep.<init>(ASRep.java:59)
	at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:60)
	... 26 more
4,435 Views
0 Kudos
7 REPLIES 7

avatar
author-rank EricL author-rank
Super Guru

Created ‎08-30-2019 06:04 PM

@Kai,

Error message "Client not found in Kerberos database" can happen due to below reasons:

- The principal listed (HTTP/host@realm for example) does not exist in the keytab
- The case of the principal/host we're connecting with does not match the case of the principal/host in the keytab (Kerberos is case sensitive)
- The principal does not exist in the KDC. Note: This sometimes happens because a principal is configured in one AD instance but you are querying another (perhaps through a VIP) and the principal has not yet been replicated.
- Multiple entries for the same principal exist in an Active Directory KDC (this breaks subsequent kinit attempts)

Please review and see if any of above could be the cause in your case.

Cheers
Eric
4,417 Views

avatar
author-rank Shelton
Master Mentor

Created ‎08-31-2019 02:00 AM

@kal 

 

whats the output of the below snippet?

# klist -kt /etc/security/keytabs/hive.service.keytab

Could you also share your krb5.conf? Please garble in the important info but not the format 🙂

4,408 Views
0 Kudos

avatar
author-rank kal
Explorer

Created on ‎08-31-2019 10:22 AM - last edited on ‎08-31-2019 05:13 PM by Moderator

Thanks for replying @Shelton @EricL .

 

We only have MIT Kerberos and doesn't have any Active Directory.

 

These are the outputs, we have two KDC setup for each cluster but they are not replicating to each other. We have one more cluster with same REALM NAME but for them also there are two KDC but there is no replication happening. Not only Hive service, even if I want to install  extra Node Manager I am getting the same error.

 

 

 

 

 

[root@spectra-xx-z15p xxxxxxx]# klist -kt /etc/security/keytabs/hive.service.keytabKeytab name: FILE:/etc/security/keytabs/hive.service.keytab
klist: Key table file '/etc/security/keytabs/hive.service.keytab' not found while starting keytab scan

 

 

 

 

 

 

 

 

[root@spectra-xx-z15p xxxxxxx]# cat /etc/krb5.conf
# Other applications require this directory to perform krb5 configuration.
includedir /etc/krb5.conf.d/
# This file is provided by the CADA client package
# Previous versions of this file can be found in /opt/cada/backups/
# $Id: krb5.conf 10925 2010-05-14 19:55:23Z xxxxxxx $


[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = SPECTRA.XXXXXXX.NET
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
forwardable = yes
renew_lifetime = 180d
[realms]
SPECTRA.XXXXXXX.NET = {
kdc = spectra-xx-z39p.sys.xxxxxxx.net
kdc = spectra-xx-z40p.sys.xxxxxxx.net
admin_server = spectra-po-z39p.sys.xxxxxxx.net
}
XXXXXXX.NET = {
kdc = kdc-m.xxxxxxx.net:88
kdc = kdc.xxxxxxx.net:88
admin_server = kdc-m.xxxxxxx.net:749
}
[domain_realm]
.xxxxxxx.net = XXXXXXX.NET
xxxxxxx.net = XXXXXXX.NET
.xxxxxxx.com = XXXXXXX.NET
xxxxxxx.com = XXXXXXX.NET
.sys.xxxxxxx.net = SPECTRA.XXXXXXX.NET
sys.xxxxxx.net = SPECTRA.xxxxxx.NET
[appdefaults]
pam = {
debug = false
forwardable = true
krb4_convert = false
chpw_prompt = sshd
}
pkinit = {
allow_pkinit = false
}

Below are the hive.keytab outputs from hive metastore and hive server.
[root@spectra-xx-z15p process]# cd /var/run/cloudera-scm-agent/process/17710-hive-HIVEMETXXTORE/
[root@spectra-xx-z15p 17710-hive-HIVEMETASTORE]# ls
cloudera-monitor.properties core-site.xml hive.keytab hive-site.xml process_timestamp sentry-site.xml yarn-conf
cloudera-stack-monitor.properties creds.localjceks hive-log4j.properties logs redaction-rules.json service-metrics.properties
[root@spectra-xx-z15p 17710-hive-HIVEMETASTORE]# klist -kt hive.keytab
Keytab name: FILE:hive.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
[root@spectra-xx-z15p 17710-hive-HIVEMETASTORE]# cd /var/run/cloudera-scm-agent/process/17709-hive-HIVESERVER2/
[root@spectra-xx-z15p 17709-hive-HIVESERVER2]# ls
cloudera-monitor.properties hive.keytab logs process_timestamp service-metrics.properties
cloudera-stack-monitor.properties hive-log4j.properties navigator.client.properties redaction-rules.json yarn-conf
core-site.xml hive-site.xml navigator.lineage.client.properties sentry-site.xml
[root@spectra-xx-z15p 17709-hive-HIVESERVER2]# klist -kt hive.keytab
Keytab name: FILE:hive.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
[root@spectra-xx-z15p 17709-hive-HIVESERVER2]#

 

4,396 Views
0 Kudos

avatar
author-rank Shelton
Master Mentor

Created ‎08-31-2019 12:35 PM

@kal 

I have written an article in response to a similar question in HCC before, you have a couple of things to do before this can work! My 2 cents advice you should have first configured the 3 clusters using single KDC, then added a second KDC's on each cluster then proceed to configure Cross_Realm Trust. In the process, you would have gained some knowledge on the implementation having said that you will need KDCs in a Master/Slave configuration to be able to propagate principal and Keytabs between the 2 KDC using krb5_prop this is another chapter on its own.


I have already noticed also an error in your krb5.conf in the below part, first you have 3 KDC's which are not replicating.there are specific steps to enable replication between KDC's see the one highlighted in ORANGE the values on the left in [lower case ] should mirror on the right in [upper case] see valid example in BLUE

 

[domain_realm]
.xxxxxxx.net = XXXXXXX.NET
xxxxxxx.net = XXXXXXX.NET
.xxxxxxx.com = XXXXXXX.NET
xxxxxxx.com = XXXXXXX.NET
.sys.xxxxxxx.net = SPECTRA.XXXXXXX.NET
sys.xxxxxx.net = SPECTRA.xxxxxx.NET

--Valid--
[domain_realm]
.xxxxxxx.net = XXXXXXX.NET
xxxxxxx.net = XXXXXXX.NET
.xxxxxxx.com = XXXXXXX.COM
xxxxxxx.com = XXXXXXX.COM
.spectra.xxxxxxx.net = SPECTRA.XXXXXXX.NET
spectra.xxxxxx.net = SPECTRA.xxxxxx.NET


You should also configure /etc/hosts file on all the 3 clusters to have IP--HOSTNAME--ALIAS and the files should be copied to all hosts in the cluster if DNS is not resolving. The example below depicts hosts in the 3 different network segments

# Cluster 1
192.168.0.1 node1.SPECTRA.XXXXXXX.NET node1
192.168.0.2 node2.SPECTRA.XXXXXXX.NET node2
....
192.168.0.3 node3.SPECTRA.XXXXXXX.NET node3

# Cluster 2
192.168.1.10 node01.XXXXXXX.NET node01
192.168.1.20 node02.XXXXXXX.NET node02
.........
192.168.1.30 node03.XXXXXXX.NET node01

# Cluster 3
192.168.2.30 nodex.XXXXXXX.COM nodex
192.168.2.40 nodey.XXXXXXX.COM nodey
.........
192.168.2.50 nodez.XXXXXXX.COM nodez

if you could breakdown your steps it would be easier to achieve as I reiterated in the beginning but its doable task.
Please let me know

4,393 Views
0 Kudos

avatar
author-rank EricL author-rank
Super Guru

Created ‎08-31-2019 09:18 PM

Does Hive use this keytab file: /etc/security/keytabs/hive.service.keytab?

I can see klist complained that the file was not found:
klist: Key table file '/etc/security/keytabs/hive.service.keytab' not found while starting keytab scan

Can you please confirm the permission of the file?

ls -al /etc/security/keytabs/hive.service.keytab

Cheers
Eric
4,377 Views
0 Kudos

avatar
author-rank Shelton
Master Mentor

Created ‎08-31-2019 11:44 PM

@EricL 

Sorry about the confusion, the location of /etc/security/keytabs/* is specific to HDP the equivalent of /var/run/cloudera-scm-agent/process   so try to map  so the command 

 

$ klist -kt  /var/run/cloudera-scm-agent/process/*.keytab   shoule be valid 

4,375 Views
0 Kudos

avatar
author-rank EricL author-rank
Super Guru

Created ‎09-01-2019 05:42 AM

Thanks @Shelton for your info.

But @kal 's command got message "Key table file '/etc/security/keytabs/hive.service.keytab' not found", so I am not sure if it is relevant.

Cheers
4,357 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements