I found the issue with our KDC server setup, We use master and slave KDC server. Hadoop was using slave KDC for authentication and Updates were made on Master but not replicated properly. While using kinit, key was working. Once we reviewed KDC krb5.log we found same message in krb log that user was not present. Once the replication issue is fixed. Hadoop also started working. ... View more

We are seeing similar issue. Everything was working fine for our test setup but now we started seeing this issue. You notice the "Client not found" it is relevent to jaas.conf, It has Server by default and it used to work but now we are seeing th default option Server but when we restart zookeepr, hdfs, hbase service it looks for Client. Since this dynamic config we can not do manual fix, we try running command manually after fix, it works for zookeeper, while there is no jaas.conf in hdfs folder.   What could have change that all applications starts looking for "Client" from "Server" option in jaas.conf.     ERROR org.apache.zookeeper.server.ZooKeeperServerMain: Unexpected exception, exiting abnormally java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:116) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)     ... View more