- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Hive Service won't start. LoginException: Client not found in Kerberos database
- Labels:
-
Apache Hive
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hive Service won't start (HiveMetaStore [main]: org.apache.thrift.transport.TTransportException: java.io.IOException: Login failure for hive/xxxx.sys.xxxx.net@REALM.NET from keytab hive.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database )
HiveMetaStore |
[main]: org.apache.thrift.transport.TTransportException: java.io.IOException: Login failure for hive/shive/xxxx.sys.xxxx.net@REALM.NET from keytab hive.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.<init>(HadoopThriftAuthBridge.java:358) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge.createServer(HadoopThriftAuthBridge.java:102) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6138) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6057) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.io.IOException: Login failure for hive/spectra-as-z15p.sys.comcast.net@SPECTRA.COMCAST.NET from keytab hive.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.<init>(HadoopThriftAuthBridge.java:353) ... 9 more Caused by: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:587) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:953) ... 10 more Caused by: KrbException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:82) at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316) at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776) ... 23 more Caused by: KrbException: Identifier doesn't match expected value (906) at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140) at sun.security.krb5.internal.ASRep.init(ASRep.java:64) at sun.security.krb5.internal.ASRep.<init>(ASRep.java:59) at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:60) ... 26 more |
Created ‎08-30-2019 06:04 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Error message "Client not found in Kerberos database" can happen due to below reasons:
- The principal listed (HTTP/host@realm for example) does not exist in the keytab
- The case of the principal/host we're connecting with does not match the case of the principal/host in the keytab (Kerberos is case sensitive)
- The principal does not exist in the KDC. Note: This sometimes happens because a principal is configured in one AD instance but you are querying another (perhaps through a VIP) and the principal has not yet been replicated.
- Multiple entries for the same principal exist in an Active Directory KDC (this breaks subsequent kinit attempts)
Please review and see if any of above could be the cause in your case.
Cheers
Eric
Created ‎08-31-2019 02:00 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
whats the output of the below snippet?
# klist -kt /etc/security/keytabs/hive.service.keytab
Could you also share your krb5.conf? Please garble in the important info but not the format 🙂
Created on
‎08-31-2019
10:22 AM
- last edited on
‎08-31-2019
05:13 PM
by
ask_bill_brooks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for replying @Shelton @EricL .
We only have MIT Kerberos and doesn't have any Active Directory.
These are the outputs, we have two KDC setup for each cluster but they are not replicating to each other. We have one more cluster with same REALM NAME but for them also there are two KDC but there is no replication happening. Not only Hive service, even if I want to install extra Node Manager I am getting the same error.
[root@spectra-xx-z15p xxxxxxx]# klist -kt /etc/security/keytabs/hive.service.keytabKeytab name: FILE:/etc/security/keytabs/hive.service.keytab
klist: Key table file '/etc/security/keytabs/hive.service.keytab' not found while starting keytab scan
[root@spectra-xx-z15p xxxxxxx]# cat /etc/krb5.conf
# Other applications require this directory to perform krb5 configuration.
includedir /etc/krb5.conf.d/
# This file is provided by the CADA client package
# Previous versions of this file can be found in /opt/cada/backups/
# $Id: krb5.conf 10925 2010-05-14 19:55:23Z xxxxxxx $
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = SPECTRA.XXXXXXX.NET
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
forwardable = yes
renew_lifetime = 180d
[realms]
SPECTRA.XXXXXXX.NET = {
kdc = spectra-xx-z39p.sys.xxxxxxx.net
kdc = spectra-xx-z40p.sys.xxxxxxx.net
admin_server = spectra-po-z39p.sys.xxxxxxx.net
}
XXXXXXX.NET = {
kdc = kdc-m.xxxxxxx.net:88
kdc = kdc.xxxxxxx.net:88
admin_server = kdc-m.xxxxxxx.net:749
}
[domain_realm]
.xxxxxxx.net = XXXXXXX.NET
xxxxxxx.net = XXXXXXX.NET
.xxxxxxx.com = XXXXXXX.NET
xxxxxxx.com = XXXXXXX.NET
.sys.xxxxxxx.net = SPECTRA.XXXXXXX.NET
sys.xxxxxx.net = SPECTRA.xxxxxx.NET
[appdefaults]
pam = {
debug = false
forwardable = true
krb4_convert = false
chpw_prompt = sshd
}
pkinit = {
allow_pkinit = false
}
Below are the hive.keytab outputs from hive metastore and hive server.
[root@spectra-xx-z15p process]# cd /var/run/cloudera-scm-agent/process/17710-hive-HIVEMETXXTORE/
[root@spectra-xx-z15p 17710-hive-HIVEMETASTORE]# ls
cloudera-monitor.properties core-site.xml hive.keytab hive-site.xml process_timestamp sentry-site.xml yarn-conf
cloudera-stack-monitor.properties creds.localjceks hive-log4j.properties logs redaction-rules.json service-metrics.properties
[root@spectra-xx-z15p 17710-hive-HIVEMETASTORE]# klist -kt hive.keytab
Keytab name: FILE:hive.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
[root@spectra-xx-z15p 17710-hive-HIVEMETASTORE]# cd /var/run/cloudera-scm-agent/process/17709-hive-HIVESERVER2/
[root@spectra-xx-z15p 17709-hive-HIVESERVER2]# ls
cloudera-monitor.properties hive.keytab logs process_timestamp service-metrics.properties
cloudera-stack-monitor.properties hive-log4j.properties navigator.client.properties redaction-rules.json yarn-conf
core-site.xml hive-site.xml navigator.lineage.client.properties sentry-site.xml
[root@spectra-xx-z15p 17709-hive-HIVESERVER2]# klist -kt hive.keytab
Keytab name: FILE:hive.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 hive/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
2 08/29/2019 16:57:43 HTTP/spectra-xx-z15p.sys.xxxxxxx.net@SPECTRA.XXXXXXX.NET
[root@spectra-xx-z15p 17709-hive-HIVESERVER2]#
Created ‎08-31-2019 12:35 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have written an article in response to a similar question in HCC before, you have a couple of things to do before this can work! My 2 cents advice you should have first configured the 3 clusters using single KDC, then added a second KDC's on each cluster then proceed to configure Cross_Realm Trust. In the process, you would have gained some knowledge on the implementation having said that you will need KDCs in a Master/Slave configuration to be able to propagate principal and Keytabs between the 2 KDC using krb5_prop this is another chapter on its own.
I have already noticed also an error in your krb5.conf in the below part, first you have 3 KDC's which are not replicating.there are specific steps to enable replication between KDC's see the one highlighted in ORANGE the values on the left in [lower case ] should mirror on the right in [upper case] see valid example in BLUE
[domain_realm]
.xxxxxxx.net = XXXXXXX.NET
xxxxxxx.net = XXXXXXX.NET
.xxxxxxx.com = XXXXXXX.NET
xxxxxxx.com = XXXXXXX.NET
.sys.xxxxxxx.net = SPECTRA.XXXXXXX.NET
sys.xxxxxx.net = SPECTRA.xxxxxx.NET
--Valid--
[domain_realm]
.xxxxxxx.net = XXXXXXX.NET
xxxxxxx.net = XXXXXXX.NET
.xxxxxxx.com = XXXXXXX.COM
xxxxxxx.com = XXXXXXX.COM
.spectra.xxxxxxx.net = SPECTRA.XXXXXXX.NET
spectra.xxxxxx.net = SPECTRA.xxxxxx.NET
You should also configure /etc/hosts file on all the 3 clusters to have IP--HOSTNAME--ALIAS and the files should be copied to all hosts in the cluster if DNS is not resolving. The example below depicts hosts in the 3 different network segments
# Cluster 1
192.168.0.1 node1.SPECTRA.XXXXXXX.NET node1
192.168.0.2 node2.SPECTRA.XXXXXXX.NET node2
....
192.168.0.3 node3.SPECTRA.XXXXXXX.NET node3
# Cluster 2
192.168.1.10 node01.XXXXXXX.NET node01
192.168.1.20 node02.XXXXXXX.NET node02
.........
192.168.1.30 node03.XXXXXXX.NET node01
# Cluster 3
192.168.2.30 nodex.XXXXXXX.COM nodex
192.168.2.40 nodey.XXXXXXX.COM nodey
.........
192.168.2.50 nodez.XXXXXXX.COM nodez
if you could breakdown your steps it would be easier to achieve as I reiterated in the beginning but its doable task.
Please let me know
Created ‎08-31-2019 09:18 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I can see klist complained that the file was not found:
klist: Key table file '/etc/security/keytabs/hive.service.keytab' not found while starting keytab scan
Can you please confirm the permission of the file?
ls -al /etc/security/keytabs/hive.service.keytab
Cheers
Eric
Created ‎08-31-2019 11:44 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Sorry about the confusion, the location of /etc/security/keytabs/* is specific to HDP the equivalent of /var/run/cloudera-scm-agent/process so try to map so the command
$ klist -kt /var/run/cloudera-scm-agent/process/*.keytab shoule be valid
Created ‎09-01-2019 05:42 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
