Support Questions

cjervis · ‎08-30-2019

Hive Service won't start (HiveMetaStore [main]: org.apache.thrift.transport.TTransportException: java.io.IOException: Login failure for hive/xxxx.sys.xxxx.net@REALM.NET from keytab hive.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database )

HiveMetaStore

[main]: org.apache.thrift.transport.TTransportException: java.io.IOException: Login failure for hive/shive/xxxx.sys.xxxx.net@REALM.NET from keytab hive.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.<init>(HadoopThriftAuthBridge.java:358)
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createServer(HadoopThriftAuthBridge.java:102)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6138)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6057)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Login failure for hive/spectra-as-z15p.sys.comcast.net@SPECTRA.COMCAST.NET from keytab hive.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962)
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.<init>(HadoopThriftAuthBridge.java:353)
	... 9 more
Caused by: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:953)
	... 10 more
Caused by: KrbException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:82)
	at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
	at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
	... 23 more
Caused by: KrbException: Identifier doesn't match expected value (906)
	at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
	at sun.security.krb5.internal.ASRep.init(ASRep.java:64)
	at sun.security.krb5.internal.ASRep.<init>(ASRep.java:59)
	at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:60)
	... 26 more

EricL · ‎08-30-2019

@Kai,

Error message "Client not found in Kerberos database" can happen due to below reasons:

- The principal listed (HTTP/host@realm for example) does not exist in the keytab
- The case of the principal/host we're connecting with does not match the case of the principal/host in the keytab (Kerberos is case sensitive)
- The principal does not exist in the KDC. Note: This sometimes happens because a principal is configured in one AD instance but you are querying another (perhaps through a VIP) and the principal has not yet been replicated.
- Multiple entries for the same principal exist in an Active Directory KDC (this breaks subsequent kinit attempts)

Please review and see if any of above could be the cause in your case.

Cheers
Eric

Shelton · ‎08-31-2019

@kal

whats the output of the below snippet?

# klist -kt /etc/security/keytabs/hive.service.keytab

Could you also share your krb5.conf? Please garble in the important info but not the format 🙂

ask_bill_brooks · ‎08-31-2019

Thanks for replying @Shelton @EricL .

We only have MIT Kerberos and doesn't have any Active Directory.

These are the outputs, we have two KDC setup for each cluster but they are not replicating to each other. We have one more cluster with same REALM NAME but for them also there are two KDC but there is no replication happening. Not only Hive service, even if I want to install extra Node Manager I am getting the same error.

[root@spectra-xx-z15p xxxxxxx]# klist -kt /etc/security/keytabs/hive.service.keytabKeytab name: FILE:/etc/security/keytabs/hive.service.keytab
klist: Key table file '/etc/security/keytabs/hive.service.keytab' not found while starting keytab scan

[root@spectra-xx-z15p xxxxxxx]# cat /etc/krb5.conf
# Other applications require this directory to perform krb5 configuration.
includedir /etc/krb5.conf.d/
# This file is provided by the CADA client package
# Previous versions of this file can be found in /opt/cada/backups/
# $Id: krb5.conf 10925 2010-05-14 19:55:23Z xxxxxxx $


[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = SPECTRA.XXXXXXX.NET
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
forwardable = yes
renew_lifetime = 180d
[realms]
SPECTRA.XXXXXXX.NET = {
kdc = spectra-xx-z39p.sys.xxxxxxx.net
kdc = spectra-xx-z40p.sys.xxxxxxx.net
admin_server = spectra-po-z39p.sys.xxxxxxx.net
}
XXXXXXX.NET = {
kdc = kdc-m.xxxxxxx.net:88
kdc = kdc.xxxxxxx.net:88
admin_server = kdc-m.xxxxxxx.net:749
}
[domain_realm]
.xxxxxxx.net = XXXXXXX.NET
xxxxxxx.net = XXXXXXX.NET
.xxxxxxx.com = XXXXXXX.NET
xxxxxxx.com = XXXXXXX.NET
.sys.xxxxxxx.net = SPECTRA.XXXXXXX.NET
sys.xxxxxx.net = SPECTRA.xxxxxx.NET
[appdefaults]
pam = {
debug = false
forwardable = true
krb4_convert = false
chpw_prompt = sshd
}
pkinit = {
allow_pkinit = false
}

Below are the hive.keytab outputs from hive metastore and hive server.
[root@spectra-xx-z15p process]# cd /var/run/cloudera-scm-agent/process/17710-hive-HIVEMETXXTORE/
[root@spectra-xx-z15p 17710-hive-HIVEMETASTORE]# ls
cloudera-monitor.properties core-site.xml hive.keytab hive-site.xml process_timestamp sentry-site.xml yarn-conf
cloudera-stack-monitor.properties creds.localjceks hive-log4j.properties logs redaction-rules.json service-metrics.properties
[root@spectra-xx-z15p 17710-hive-HIVEMETASTORE]# klist -kt hive.keytab
Keytab name: FILE:hive.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
[root@spectra-xx-z15p 17710-hive-HIVEMETASTORE]# cd /var/run/cloudera-scm-agent/process/17709-hive-HIVESERVER2/
[root@spectra-xx-z15p 17709-hive-HIVESERVER2]# ls
cloudera-monitor.properties hive.keytab logs process_timestamp service-metrics.properties
cloudera-stack-monitor.properties hive-log4j.properties navigator.client.properties redaction-rules.json yarn-conf
core-site.xml hive-site.xml navigator.lineage.client.properties sentry-site.xml
[root@spectra-xx-z15p 17709-hive-HIVESERVER2]# klist -kt hive.keytab
Keytab name: FILE:hive.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
 2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
[root@spectra-xx-z15p 17709-hive-HIVESERVER2]#

Shelton · ‎08-31-2019

@kal

I have written an article in response to a similar question in HCC before, you have a couple of things to do before this can work! My 2 cents advice you should have first configured the 3 clusters using single KDC, then added a second KDC's on each cluster then proceed to configure Cross_Realm Trust. In the process, you would have gained some knowledge on the implementation having said that you will need KDCs in a Master/Slave configuration to be able to propagate principal and Keytabs between the 2 KDC using krb5_prop this is another chapter on its own.

I have already noticed also an error in your krb5.conf in the below part, first you have 3 KDC's which are not replicating.there are specific steps to enable replication between KDC's see the one highlighted in ORANGE the values on the left in [lower case ] should mirror on the right in [upper case] see valid example in BLUE

[domain_realm]
.xxxxxxx.net = XXXXXXX.NET
xxxxxxx.net = XXXXXXX.NET
.xxxxxxx.com = XXXXXXX.NET
xxxxxxx.com = XXXXXXX.NET
.sys.xxxxxxx.net = SPECTRA.XXXXXXX.NET
sys.xxxxxx.net = SPECTRA.xxxxxx.NET

--Valid--
[domain_realm]
.xxxxxxx.net = XXXXXXX.NET
xxxxxxx.net = XXXXXXX.NET
.xxxxxxx.com = XXXXXXX.COM
xxxxxxx.com = XXXXXXX.COM
.spectra.xxxxxxx.net = SPECTRA.XXXXXXX.NET
spectra.xxxxxx.net = SPECTRA.xxxxxx.NET

You should also configure /etc/hosts file on all the 3 clusters to have IP--HOSTNAME--ALIAS and the files should be copied to all hosts in the cluster if DNS is not resolving. The example below depicts hosts in the 3 different network segments

# Cluster 1
192.168.0.1 node1.SPECTRA.XXXXXXX.NET node1
192.168.0.2 node2.SPECTRA.XXXXXXX.NET node2
....
192.168.0.3 node3.SPECTRA.XXXXXXX.NET node3

# Cluster 2
192.168.1.10 node01.XXXXXXX.NET node01
192.168.1.20 node02.XXXXXXX.NET node02
.........
192.168.1.30 node03.XXXXXXX.NET node01

# Cluster 3
192.168.2.30 nodex.XXXXXXX.COM nodex
192.168.2.40 nodey.XXXXXXX.COM nodey
.........
192.168.2.50 nodez.XXXXXXX.COM nodez

if you could breakdown your steps it would be easier to achieve as I reiterated in the beginning but its doable task.
Please let me know

EricL · ‎08-31-2019

Does Hive use this keytab file: /etc/security/keytabs/hive.service.keytab?

I can see klist complained that the file was not found:
klist: Key table file '/etc/security/keytabs/hive.service.keytab' not found while starting keytab scan

Can you please confirm the permission of the file?

ls -al /etc/security/keytabs/hive.service.keytab

Cheers
Eric

Shelton · ‎08-31-2019

@EricL

Sorry about the confusion, the location of /etc/security/keytabs/* is specific to HDP the equivalent of /var/run/cloudera-scm-agent/process so try to map so the command

$ klist -kt /var/run/cloudera-scm-agent/process/*.keytab shoule be valid

EricL · ‎09-01-2019

Thanks @Shelton for your info.

But @kal 's command got message "Key table file '/etc/security/keytabs/hive.service.keytab' not found", so I am not sure if it is relevant.

Cheers

Cloudera Community

Support Questions

Hive Service won't start. LoginException: Client not found in Kerberos database

404 service kerberos not found

hadoop client kerberos error

Backup and Restore HDP Kerberos Database

Connecting DbVisualizer and DataGrip to Hive with ...

Bad : The Hive Metastore canary failed to create a...

Hive service doesn't start

Some services failing to come up after enabling ke...

Storm Service Check Error - Kerberos: "The TGT fou...

Max number of databases and tables allowed in Hive...

Azure Data Factory Linked service to CDP Hive Kerb...