Ambari : Strange kerberos authentication error

Question

recently ambari server logs are showing some warnings

/var/log/ambari-server/ambari-server.log

02 Jul 2021 18:43:52,514  INFO [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:109 - Translated knox/<knox_gateway>@<REALM> to knox using auth-to-local rules during Kerberos authentication.
02 Jul 2021 18:43:52,515  WARN [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:143 - Failed find user account for user with username of knox during Kerberos authentication.
02 Jul 2021 18:43:52,516  WARN [ambari-client-thread-792188] AmbariKerberosAuthenticationFilter:149 - Negotiate Header was invalid: Negotiate YIIDlAYGKwYBBQUCoIIDiDCCA4SgDTALBgkqhkiG9xIBAgKhBAMCAXaiggNrBIIDZ2CCA2MGCSqGSIb3EgECAgEAboIDUjCCA06gAwIBBaEDAgEOogcDBQAgAAAAo4IB/WGCAfkwggH1oAMCAQWhMxsxMjZGNURFMDEtNUU0MC00RDhBLTk4QkQtQTQzNTNCN0JGNUUzLkRBVEFMQUtFLk9WSKJPME2gAwIBAKFGMEQbBEhUVFAbPG92aC1lbm9kZTYuMjZmNWRlMDEtNWU0MC00ZDhhLTk4YmQtYTQzNTNiN2JmNWUzLmRhdGFsYWtlLm92aKOCAWYwggFioAMCARKhAwIBAqKCAVQEggFQbqKii/FFb+dRh+NYor9rVsacj0GwwyOV1KCZLEFkyM7VEhjK/wV8gkJOL9V0u99lkAIPGFCI6TzHenX6VtiZEd/MvKox5b4V0REwnpTuX5vS5lXCfXtbEOF2SbXmch1/U5RCjVQr8+vm/8AYmILIsqnFFmU9AgFFtEtZsH1NZpiSbosJs3oDJ1yINbzuzrxpCJJTu2Rcv2j34mN8+SkA42D6ZO6yP1iwlTkWji9VEDSOYLdzzhPMq0rmUCuuHfDYHX2Dx09xx4h9C6d28E3+o07LmUTUOBbnkiwnk57HZ+muxfD7T93PHOcfcJGPPKFWnUzTBg/ZrNsU3M36Y6UdqJ3kH/nvu6TNZDy+EbJhztlz3H1xNlMQ7D9np+HNHd72CvgFFzEbLCqkR9WUUL0R9WIno10V2wX8gBXPV31qY+WU59vQHfa9kkmytR8a60sYpIIBNjCCATKgAwIBEqKCASkEggEle7R2DYTiWwLkpH4BKwKGmoZfD7ZX0garqKPDQ7HDBDJIXUTce4UELJ84bTwn/DtpnUeQ0rZ70FedqHlHutB60NsswXvzoozLk+7Meeo3c310iWNIEd++dtEo0ZUc4sE8CA984VPYHHral67L/L4rQgb39ikYhxo9ieLMMHZ7Im6uzsnQzH0shKSHgWsroneqSvQXAuNM+08OAWuP0znVFAddSpPiQYlnsf9LJcjrn0hLFQwoFV48HA7/PyGDZ8Jotw/UMD/sbokOazOvzubvGdoyS6+8xJlJnf+D9WneoviqOFn4Po6B8WANQuF02QbPOPeE4XKXA4n6iruZmdKeR1ksx6OuEGygRSs1lmsgOzZvrjNLCay2ty4qtsMUUCjQz9V8c4A=
org.springframework.security.core.userdetails.UsernameNotFoundException: Failed find user account for user with username of knox during Kerberos authentication.
        at org.apache.ambari.server.security.authentication.kerberos.AmbariAuthToLocalUserDetailsService.createUser(AmbariAuthToLocalUserDetailsService.java:144)
        at org.apache.ambari.server.security.authentication.kerberos.AmbariAuthToLocalUserDetailsService.loadUserByUsername(AmbariAuthToLocalUserDetailsService.java:110)
        at org.springframework.security.kerberos.authentication.KerberosServiceAuthenticationProvider.authenticate(KerberosServiceAuthenticationProvider.java:66)
        at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:156)
        at org.springframework.security.kerberos.web.authentication.SpnegoAuthenticationProcessingFilter.doFilter(SpnegoAuthenticationProcessingFilter.java:145)
        at org.apache.ambari.server.security.authentication.kerberos.AmbariKerberosAuthenticationFilter.doFilter(AmbariKerberosAuthenticationFilter.java:167)
        at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:120)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212)
        at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201)
        at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:745)

/var/log/ambari-server/ambari-audit.log

2021-07-02T19:01:16.881+0200, User(null), RemoteIp(xxx.xxx.xxx.xxx), Operation(User login), Roles(
), Status(Failed), Reason(Failed find user account for user with username of knox during Kerberos authentication.)

There is no identified issue right now, but it generate huge logs (50 lines/s) and we are unable to read ambari-logs

Ambari version : 2.6.2.2

HDP Version : HDP-2.6.5.1100

enirys · Accepted Answer

I finaly found the root cause of this issue, it happens when users connect to ambari web ui and the stay connected for a while then the session is killed due to timeout.

In my configuration there are no timeout for :

user.inactivity.timeout.default, user.inactivity.timeout.role.readonly.default

Setting these properties allows ambari to logout users after a period of inactivity.

This error may occurs also when we restart the ambari-server, users still connected with a 401 error message

i'm closing this thread and opening a new one with more details about the issue

https://community.cloudera.com/t5/Support-Questions/Logout-ambari-s-connected-users-on-ambari-server-restart/td-p/321322

pvishnu · Answer

Hi @enirys ,

It appears that you have configured Ambari to authenticate using Kerberos tokens via SPNEGO.

From the logs, I see you are trying to authenticate to Ambari using the principal "knox/<knox_gateway>@<REALM>" which gets translated to the user name "knox" and then searches it in the internal database or from an external source, such as an LDAP directory based on your configuration and unable to find it and hence the below warning message,

----------

02 Jul 2021 18:43:52,515 WARN [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:143 - Failed find user account for user with username of knox during Kerberos authentication.

----------

So validate the values set for the property "authentication.kerberos.user.types", and ensure the "knox" user is present in the User type mentioned and create if unavailable. Once available and when you re-authenticate again, Ambari will be able to find the relevant user and it bypasses the default user name and password login facility and should be able to authenticate successfully.

Let us know if this helps!

Thanks,
Prashanth Vishnu

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.