question Re: Ambari : Strange kerberos authentication error in Support Questions

Ambari : Strange kerberos authentication error

enirys — Fri, 02 Jul 2021 17:04:57 GMT

recently ambari server logs are showing some warnings

/var/log/ambari-server/ambari-server.log

02 Jul 2021 18:43:52,514 INFO [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:109 - Translated knox/<knox_gateway>@<REALM> to knox using auth-to-local rules during Kerberos authentication. 02 Jul 2021 18:43:52,515 WARN [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:143 - Failed find user account for user with username of knox during Kerberos authentication. 02 Jul 2021 18:43:52,516 WARN [ambari-client-thread-792188] AmbariKerberosAuthenticationFilter:149 - Negotiate Header was invalid: Negotiate YIIDlAYGKwYBBQUCoIIDiDCCA4SgDTALBgkqhkiG9xIBAgKhBAMCAXaiggNrBIIDZ2CCA2MGCSqGSIb3EgECAgEAboIDUjCCA06gAwIBBaEDAgEOogcDBQAgAAAAo4IB/WGCAfkwggH1oAMCAQWhMxsxMjZGNURFMDEtNUU0MC00RDhBLTk4QkQtQTQzNTNCN0JGNUUzLkRBVEFMQUtFLk9WSKJPME2gAwIBAKFGMEQbBEhUVFAbPG92aC1lbm9kZTYuMjZmNWRlMDEtNWU0MC00ZDhhLTk4YmQtYTQzNTNiN2JmNWUzLmRhdGFsYWtlLm92aKOCAWYwggFioAMCARKhAwIBAqKCAVQEggFQbqKii/FFb+dRh+NYor9rVsacj0GwwyOV1KCZLEFkyM7VEhjK/wV8gkJOL9V0u99lkAIPGFCI6TzHenX6VtiZEd/MvKox5b4V0REwnpTuX5vS5lXCfXtbEOF2SbXmch1/U5RCjVQr8+vm/8AYmILIsqnFFmU9AgFFtEtZsH1NZpiSbosJs3oDJ1yINbzuzrxpCJJTu2Rcv2j34mN8+SkA42D6ZO6yP1iwlTkWji9VEDSOYLdzzhPMq0rmUCuuHfDYHX2Dx09xx4h9C6d28E3+o07LmUTUOBbnkiwnk57HZ+muxfD7T93PHOcfcJGPPKFWnUzTBg/ZrNsU3M36Y6UdqJ3kH/nvu6TNZDy+EbJhztlz3H1xNlMQ7D9np+HNHd72CvgFFzEbLCqkR9WUUL0R9WIno10V2wX8gBXPV31qY+WU59vQHfa9kkmytR8a60sYpIIBNjCCATKgAwIBEqKCASkEggEle7R2DYTiWwLkpH4BKwKGmoZfD7ZX0garqKPDQ7HDBDJIXUTce4UELJ84bTwn/DtpnUeQ0rZ70FedqHlHutB60NsswXvzoozLk+7Meeo3c310iWNIEd++dtEo0ZUc4sE8CA984VPYHHral67L/L4rQgb39ikYhxo9ieLMMHZ7Im6uzsnQzH0shKSHgWsroneqSvQXAuNM+08OAWuP0znVFAddSpPiQYlnsf9LJcjrn0hLFQwoFV48HA7/PyGDZ8Jotw/UMD/sbokOazOvzubvGdoyS6+8xJlJnf+D9WneoviqOFn4Po6B8WANQuF02QbPOPeE4XKXA4n6iruZmdKeR1ksx6OuEGygRSs1lmsgOzZvrjNLCay2ty4qtsMUUCjQz9V8c4A= org.springframework.security.core.userdetails.UsernameNotFoundException: Failed find user account for user with username of knox during Kerberos authentication. at org.apache.ambari.server.security.authentication.kerberos.AmbariAuthToLocalUserDetailsService.createUser(AmbariAuthToLocalUserDetailsService.java:144) at org.apache.ambari.server.security.authentication.kerberos.AmbariAuthToLocalUserDetailsService.loadUserByUsername(AmbariAuthToLocalUserDetailsService.java:110) at org.springframework.security.kerberos.authentication.KerberosServiceAuthenticationProvider.authenticate(KerberosServiceAuthenticationProvider.java:66) at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:156) at org.springframework.security.kerberos.web.authentication.SpnegoAuthenticationProcessingFilter.doFilter(SpnegoAuthenticationProcessingFilter.java:145) at org.apache.ambari.server.security.authentication.kerberos.AmbariKerberosAuthenticationFilter.doFilter(AmbariKerberosAuthenticationFilter.java:167) at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:120) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201) at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745)

/var/log/ambari-server/ambari-audit.log

2021-07-02T19:01:16.881+0200, User(null), RemoteIp(xxx.xxx.xxx.xxx), Operation(User login), Roles( ), Status(Failed), Reason(Failed find user account for user with username of knox during Kerberos authentication.)

There is no identified issue right now, but it generate huge logs (50 lines/s) and we are unable to read ambari-logs

Ambari version : 2.6.2.2

HDP Version : HDP-2.6.5.1100

Re: Ambari : Strange kerberos authentication error

pvishnu — Mon, 05 Jul 2021 16:42:38 GMT

Hi @enirys ,

It appears that you have configured Ambari to authenticate using Kerberos tokens via SPNEGO.

From the logs, I see you are trying to authenticate to Ambari using the principal "knox/<knox_gateway>@<REALM>" which gets translated to the user name "knox" and then searches it in the internal database or from an external source, such as an LDAP directory based on your configuration and unable to find it and hence the below warning message,

----------

02 Jul 2021 18:43:52,515  WARN [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:143 - Failed find user account for user with username of knox during Kerberos authentication.

----------

So validate the values set for the property "authentication.kerberos.user.types", and ensure the "knox" user is present in the User type mentioned and create if unavailable. Once available and when you re-authenticate again, Ambari will be able to find the relevant user and it bypasses the default user name and password login facility and should be able to authenticate successfully.

Let us know if this helps!

Thanks,
Prashanth Vishnu

Re: Ambari : Strange kerberos authentication error

enirys — Mon, 05 Jul 2021 18:10:39 GMT

Hi @pvishnu

The connection to ambari is done over knox gateway, and we have many clients using the gateway so i don't know who is causing this trouble

We have the same configuration on a dev cluster without having this issue except knox and ambari are not on the same node

as it use knox to authenticate i have created a knox/<hostname> principal on ambari node but without success, just duplicate number of logs and a new error appears in ambari-audit.log

we are using freeipa server to manage users and kerbers

2021-07-05T19:48:23.518+0200, User(null), RemoteIp(xxx.xxx.xxx.xxx), Operation(User login), Roles( ), Status(Failed), Reason(Authentication required)

i searched the property authentication.kerberos.user.types but can't find it on any component

Re: Ambari : Strange kerberos authentication error

pvishnu — Tue, 06 Jul 2021 07:21:05 GMT

Hi @enirys,

If you wanted to use the Authentication provider used with the Knox gateway for the Ambari UI and don't prefer the kerberos authentication, then you can disable it by setting the below property to false,

authentication.kerberos.enabled = false

You can find the "authentication.kerberos.*" properties in the /etc/ambari-server/conf/ambari.properties file.

Restart Ambari server after making necessary changes and monitor the logs to see if the warnings are repeated again.

Re: Ambari : Strange kerberos authentication error

enirys — Tue, 06 Jul 2021 07:41:54 GMT

Hi @pvishnu

I wan't to keep kerberos authentication, below ambari config

agent.package.install.task.timeout=36000 agent.stack.retry.on_repo_unavailability=false agent.stack.retry.tries=5 agent.task.timeout=2000 agent.threadpool.size.max=25 ambari-server.user=root ambari.ldap.isConfigured=true ambari.post.user.creation.hook=/var/lib/ambari-server/resources/scripts/post-user-creation-hook.sh ambari.post.user.creation.hook.enabled=true ambari.python.wrap=ambari-python-wrap authentication.kerberos.auth_to_local.rules=DEFAULT authentication.kerberos.enabled=true authentication.kerberos.spnego.keytab.file=/etc/security/keytabs/spnego.service.keytab authentication.kerberos.spnego.principal=HTTP/<ambari_host_fqdn> authentication.kerberos.user.types=LDAP authentication.ldap.baseDn=cn=accounts,dc=<domain>,dc=<domain>,dc=<domain> authentication.ldap.bindAnonymously=false authentication.ldap.dnAttribute=dn authentication.ldap.groupMembershipAttr=member authentication.ldap.groupNamingAttr=cn authentication.ldap.groupObjectClass=posixGroup authentication.ldap.managerDn=uid=ldapbind,cn=sysaccounts,cn=etc,dc=<domain>,dc=<domain>,dc=<domain> authentication.ldap.managerPassword=/etc/ambari-server/conf/ldap-password.dat authentication.ldap.primaryUrl=<ipa_host_fqdn>:636 authentication.ldap.useSSL=true authentication.ldap.userObjectClass=posixAccount authentication.ldap.usernameAttribute=uid

I just need to know the origin of previous warning and remove them, i suspect ambari views which are used by some users but i'm not sure.

Re: Ambari : Strange kerberos authentication error

enirys — Fri, 09 Jul 2021 15:32:52 GMT

@pvishnu

Any update please ? do you need more informations ?

Re: Ambari : Strange kerberos authentication error

pvishnu — Tue, 13 Jul 2021 19:07:31 GMT

@enirys , Try checking the Ambari audit logs to identify where the authentication requests using the Knox kerberos principal is originating from.

Re: Ambari : Strange kerberos authentication error

enirys — Thu, 22 Jul 2021 10:00:59 GMT

I finaly found the root cause of this issue, it happens when users connect to ambari web ui and the stay connected for a while then the session is killed due to timeout.

In my configuration there are no timeout for :

user.inactivity.timeout.default, user.inactivity.timeout.role.readonly.default

Setting these properties allows ambari to logout users after a period of inactivity.

This error may occurs also when we restart the ambari-server, users still connected with a 401 error message

i'm closing this thread and opening a new one with more details about the issue

https://community.cloudera.com/t5/Support-Questions/Logout-ambari-s-connected-users-on-ambari-server-restart/td-p/321322