[varnika@ambaristandby ~]$ hdfs dfs -ls hdfs://172.21.58.115:8020/user/
17/03/29 04:26:18 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
17/03/29 04:26:21 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
17/03/29 04:26:25 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
17/03/29 04:26:27 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
17/03/29 04:26:31 WARN ipc.Client: Couldn't setup connection for varnika@EXAMPLE.COM to /172.21.58.115:8020
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
        at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757)
        at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
        at org.apache.hadoop.ipc.Client.call(Client.java:1449)
        at org.apache.hadoop.ipc.Client.call(Client.java:1396)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
        at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:816)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
        at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2158)
        at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1423)
        at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1419)
        at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
        at org.apache.hadoop.fs.Globber.glob(Globber.java:252)
        at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1674)
        at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
        at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
        at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
        at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
        at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:297)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:350)
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
Caused by: KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
        at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192)
        at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203)
        at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309)
        at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115)
        at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
        ... 44 more
Caused by: KrbException: Identifier doesn't match expected value (906)
        at sun.security.krb5.internal.KDCRep.init(KDCRep.java:143)
        at sun.security.krb5.internal.TGSRep.init(TGSRep.java:66)
        at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:61)
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
        ... 50 more
17/03/29 04:26:31 WARN retry.RetryInvocationHandler: Exception while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over null. Not retrying because try once and fail.
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for varnika@EXAMPLE.COM to /172.21.58.115:8020; Host Details : local host is: "ambaristandby.myhadoop.com/172.21.58.116"; destination host is: "ms.myhadoop.com":8020;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
        at org.apache.hadoop.ipc.Client.call(Client.java:1496)
        at org.apache.hadoop.ipc.Client.call(Client.java:1396)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
        at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:816)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
        at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2158)
        at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1423)
        at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1419)
        at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
        at org.apache.hadoop.fs.Globber.glob(Globber.java:252)
        at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1674)
        at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
        at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
        at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
        at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
        at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:297)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:350)
Caused by: java.io.IOException: Couldn't setup connection for varnika@EXAMPLE.COM to /172.21.58.115:8020
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:712)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:683)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:770)
        at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
        at org.apache.hadoop.ipc.Client.call(Client.java:1449)
        ... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
        at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757)
        ... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
        ... 41 more
Caused by: KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
        at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192)
        at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203)
        at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309)
        at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115)
        at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
        ... 44 more
Caused by: KrbException: Identifier doesn't match expected value (906)
        at sun.security.krb5.internal.KDCRep.init(KDCRep.java:143)
        at sun.security.krb5.internal.TGSRep.init(TGSRep.java:66)
        at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:61)
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
        ... 50 more
ls: Failed on local exception: java.io.IOException: Couldn't setup connection for varnika@EXAMPLE.COM to /172.21.58.115:8020; Host Details : local host is: "ambaristandby.myhadoop.com/172.21.58.116"; destination host is: "ms.myhadoop.com":8020;



===================================/var/log/krb5kdc.log==================================================================

Server:ambaristandby.myhadoop.com

Mar 29 05:20:21 ambaristandby.myhadoop.com krb5kdc[1177](info): AS_REQ (4 etypes {18 17 16 23}) 172.21.58.120: ISSUE: authtime 1490757621, etypes {rep=18 tkt=18 ses=18}, HTTP/standbyms.myhadoop.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Mar 29 05:20:21 ambaristandby.myhadoop.com krb5kdc[1177](info): AS_REQ (4 etypes {18 17 16 23}) 172.21.58.120: ISSUE: authtime 1490757621, etypes {rep=18 tkt=18 ses=18}, yarn/standbyms.myhadoop.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Mar 29 05:20:21 ambaristandby.myhadoop.com krb5kdc[1177](info): AS_REQ (4 etypes {18 17 16 23}) 172.21.58.120: ISSUE: authtime 1490757621, etypes {rep=18 tkt=18 ses=18}, rm/standbyms.myhadoop.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Mar 29 05:20:21 ambaristandby.myhadoop.com krb5kdc[1177](info): AS_REQ (4 etypes {18 17 16 23}) 172.21.58.120: ISSUE: authtime 1490757621, etypes {rep=18 tkt=18 ses=18}, zookeeper/standbyms.myhadoop.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Mar 29 05:21:07 ambaristandby.myhadoop.com krb5kdc[1177](info): TGS_REQ (6 etypes {18 17 16 23 1 3}) 172.21.58.116: UNKNOWN_SERVER: authtime 0,  varnika@EXAMPLE.COM for nn/ms.myhadoop.com@EXAMPLE.COM, Server not found in Kerberos database
Mar 29 05:21:08 ambaristandby.myhadoop.com krb5kdc[1177](info): TGS_REQ (6 etypes {18 17 16 23 1 3}) 172.21.58.116: UNKNOWN_SERVER: authtime 0,  varnika@EXAMPLE.COM for nn/ms.myhadoop.com@EXAMPLE.COM, Server not found in Kerberos database
Mar 29 05:21:09 ambaristandby.myhadoop.com krb5kdc[1177](info): TGS_REQ (6 etypes {18 17 16 23 1 3}) 172.21.58.116: UNKNOWN_SERVER: authtime 0,  varnika@EXAMPLE.COM for nn/ms.myhadoop.com@EXAMPLE.COM, Server not found in Kerberos database
Mar 29 05:21:09 ambaristandby.myhadoop.com krb5kdc[1177](info): TGS_REQ (6 etypes {18 17 16 23 1 3}) 172.21.58.116: UNKNOWN_SERVER: authtime 0,  varnika@EXAMPLE.COM for nn/ms.myhadoop.com@EXAMPLE.COM, Server not found in Kerberos database
Mar 29 05:21:10 ambaristandby.myhadoop.com krb5kdc[1177](info): TGS_REQ (6 etypes {18 17 16 23 1 3}) 172.21.58.116: UNKNOWN_SERVER: authtime 0,  varnika@EXAMPLE.COM for nn/ms.myhadoop.com@EXAMPLE.COM, Server not found in Kerberos database
Mar 29 05:21:14 ambaristandby.myhadoop.com krb5kdc[1177](info): TGS_REQ (6 etypes {18 17 16 23 1 3}) 172.21.58.116: UNKNOWN_SERVER: authtime 0,  varnika@EXAMPLE.COM for nn/ms.myhadoop.com@EXAMPLE.COM, Server not found in Kerberos database