Reply
Explorer
Posts: 6
Registered: ‎07-25-2015

javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_Fou

Hi,

I am trying to implement Kerberos security on cloudera CDH-5.3.  In kerberos implementation wizard it generates principals for all the services.

The prinicpal generated are as follows - 

 


kadmin.local: listprincs
HTTP/01hw310845.India.ABC.com@INDIA.ABC.COM
K/M@INDIA.ABC.COM
cloudera-scm@INDIA.ABC.COM
hdfs/01hw310845.India.ABC.com@INDIA.ABC.COM
hive/01hw310845.India.ABC.com@INDIA.ABC.COM
hue/01hw310845.India.ABC.com@INDIA.ABC.COM
impala/01hw310845.India.ABC.com@INDIA.ABC.COM
kadmin/01hw310845.india.ABC.com@INDIA.ABC.COM
kadmin/admin@INDIA.ABC.COM
kadmin/changepw@INDIA.ABC.COM
krbtgt/INDIA.ABC.COM@INDIA.ABC.COM
mapred/01hw310845.India.ABC.com@INDIA.ABC.COM
yarn/01hw310845.India.ABC.com@INDIA.ABC.COM
zookeeper/01hw310845.India.ABC.com@INDIA.ABC.COM

 

But when i try to start all the services in the cluster it gives following error -

 

Failed to start namenode.
java.io.IOException: Login failure for hdfs/01hw310845.india.abc.com@INDIA.ABC.COM from keytab hdfs.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:947)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:560)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:579)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:754)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:738)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1427)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1493)
Caused by: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:763)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:938)
... 7 more
Caused by: KrbException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:82)
at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:319)
at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:364)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:735)
... 20 more
Caused by: KrbException: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:143)
at sun.security.krb5.internal.ASRep.init(ASRep.java:65)
at sun.security.krb5.internal.ASRep.<init>(ASRep.java:60)
at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:60)
... 23 more

 

The problem seems to be that principal name that cloudera uses to authenticate is in SMALL LETTERS of FQDN while the generated princpals are

in CAPITAL LETTERS.

 

How to ensure that cloudera generates the principals(domain name) from - /etc/host file without converting it into small case

Posts: 1,044
Topics: 1
Kudos: 262
Solutions: 130
Registered: ‎04-22-2014

Re: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT

Hadoop in general expects that your hostnames and domain names are all lowercase.  When Kerberos is introduced, this becomes important.  While it is possible to override this behavior (of expecting lowercase) by doing manual configuration, I recommend ensuring via /etc/hosts or DNS that your host and domain are lower case.  After that is corrected, regenerate credentials and that should correct the problem.

 

Regards,

 

Ben

New Contributor
Posts: 2
Registered: ‎07-06-2016

Re: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT

We are seeing similar issue. Everything was working fine for our test setup but now we started seeing this issue.

You notice the "Client not found" it is relevent to jaas.conf, It has Server by default and it used to work but now we are seeing th default option Server but when we restart zookeepr, hdfs, hbase service it looks for Client. Since this dynamic config we can not do manual fix, we try running command manually after fix, it works for zookeeper, while there is no jaas.conf in hdfs folder.

 

What could have change that all applications starts looking for "Client" from "Server" option in jaas.conf.

 

 

ERROR org.apache.zookeeper.server.ZooKeeperServerMain: Unexpected exception, exiting abnormally
java.io.IOException: Could not configure server because SASL configuration did not allow the  ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND
	at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)
	at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:116)
	at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
	at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

 

 

Highlighted
New Contributor
Posts: 4
Registered: ‎06-13-2016

Re: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT

I have the same issue , 

 

My hosts in cluster have hostname something like this

 

192.168.X.X  Master

192.168.X.X Slave1

192.168.X.X Slave2

192.168.X.X Slave3 

 

And generated principal names were like

 

hdfs/Master@CLOUDERA

spark/Slave1@CLOUDERA

 

And when a data node is started it was looking for hdfs/master@Former Member instead of hdfs/Master@Former Member

 

Resoultion steps:

 

1)Change HOSTNAME in /etc/sysconfig/network 

 

HOSTNAME=master on Master node , HOSTNAME=slave1 on Slave1 node

2)Have all the hosts in cluster maintain same hostname 

 

192.168.X.X master

192.168.X.X slave

 

3) Reboot all hosts

 

4) Check for the hostname 

 

5) On Cloudera manager -> For each hosts - > regenrate keytab

 

6) Go to Administration->Security->KerberosCredentials ->Check the prinicipal names are with correct hosts like 

 

hdfs/master@CLOUDERA

hdfs/slave1@CLOUDERA

 

 

New Contributor
Posts: 4
Registered: ‎07-13-2016

Re: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT

Thanks Nitesh, this should be a mandatory steps in the doc to install multi datanode cluster with Kerberos enabled

New Contributor
Posts: 2
Registered: ‎07-06-2016

Re: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT

I found the issue with our KDC server setup, We use master and slave KDC server. Hadoop was using slave KDC for authentication and Updates were made on Master but not replicated properly. While using kinit, key was working. Once we reviewed KDC krb5.log we found same message in krb log that user was not present. Once the replication issue is fixed. Hadoop also started working.
Posts: 1,004
Kudos: 119
Solutions: 62
Registered: ‎04-06-2015

Re: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT

There is some great discussion here.  @singhuda have you resolved the original issue?



Cy Jervis, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

New Contributor
Posts: 3
Registered: ‎05-16-2016

Re: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT

Hi,

I modified the HOSTNAME to lowercase and modified /etc/hosts and rebooted the servers.

Cloudera Manager is generating only 3 principals (hue, solr and HTTP). If I click "generate missing credentials", it says

"No roles required Kerberos credentials to be generated."

 

Please help what needs to be done to generate all the credentials.

 

Thanks

V

krb
New Contributor
Posts: 5
Registered: ‎07-25-2018

Re: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT

I am also facing similar issue. Without kerberos all the services are running properly but when I try to kerberized the cluster with AD external authentication, the CM's wizard took me properly until stopping the cluster but when the cluster is restarting I am facing the issues in first step of hdfs dependency .. zookeeper

Unexpected exception, exiting abnormally
java.io.IOException: Could not configure server because SASL configuration did not allow the  ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Client not found in Kerberos database (6)
	at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:135)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

 we have generated credentials for common/single user for entire cluster services.. 

Any idea whats the issue..

Posts: 1,044
Topics: 1
Kudos: 262
Solutions: 130
Registered: ‎04-22-2014

Re: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT

@krb,

 

Make sure you have your /etc/krb5.conf configured correctly so that the zookeeper is sending its AS_REQ to the right KDC.  If you have just changed from one KDC to another, the /etc/krb5.conf also needs to be updated.  If you are not managing it with Cloudera Manager, it needs to be changed manually.

 

Either way, you could do a tcpdump on port 88 and check if output requests are going to the right KDC if /etc/krb5.conf is configured properly for your new KDC.

 

 

Announcements