Logout ambari's connected users on ambari-server restart

[#1]

Our ambari webui is encountering some errors when ambari-server restart. When some users are connected and we restart the ambari-server, it doesn't logout connected users and 401 errors appears

This behaviour is very constraining because it generate lot of lines in ambari logs (Thread)

How to force ambari to log out all connected users when ambari-server restart ?

[#2]

@enirys 

- Can you confirm the value you have set for user.inactivity.timeout.default, user.inactivity.timeout.role.readonly.default

- Can you share the complete 401 error users are experiencing?

- Can you confirm if you have spnego authentication enabled for ambari?

- Attach the below logs after 401 error occurs in ambari UI

1. ambari.properties file
2. ambari-audit.log
3. ambari-server.log

[#3]

@Raamar 

Yes, I'm using spnego authentication with user inactivity properties

user.inactivity.timeout.default=600
user.inactivity.timeout.role.readonly.default=300

My ambari is behind a loadbalancer (nginx), bellow the 401 error logs :

/var/log/nginx/access.log

"GET /gateway/default/ambari/api/v1/clusters/prod/requests?to=end&page_size=10&fields=Requests&_=1625812254413 HTTP/1.1" 401 51 "https://knox.26f5de01-5e40-4d8a-98bd-a4353b7bf5e3.datalake/gateway/default/ambari" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0" "-"

ambari.properties

agent.package.install.task.timeout=36000
agent.stack.retry.on_repo_unavailability=false
agent.stack.retry.tries=5
agent.task.timeout=2000
agent.threadpool.size.max=25
ambari-server.user=root
ambari.ldap.isConfigured=true
ambari.post.user.creation.hook=/var/lib/ambari-server/resources/scripts/post-user-creation-hook.sh
ambari.post.user.creation.hook.enabled=true
ambari.python.wrap=ambari-python-wrap
authentication.kerberos.auth_to_local.rules=DEFAULT
authentication.kerberos.enabled=true
authentication.kerberos.spnego.keytab.file=/etc/security/keytabs/spnego.service.keytab
authentication.kerberos.spnego.principal=HTTP/<ambari_host_fqdn>
authentication.kerberos.user.types=LDAP
authentication.ldap.baseDn=cn=accounts,dc=<domain>,dc=<domain>,dc=<domain>
authentication.ldap.bindAnonymously=false
authentication.ldap.dnAttribute=dn
authentication.ldap.groupMembershipAttr=member
authentication.ldap.groupNamingAttr=cn
authentication.ldap.groupObjectClass=posixGroup
authentication.ldap.managerDn=uid=ldapbind,cn=sysaccounts,cn=etc,dc=<domain>,dc=<domain>,dc=<domain>
authentication.ldap.managerPassword=/etc/ambari-server/conf/ldap-password.dat
authentication.ldap.primaryUrl=<ipa_host_fqdn>:636
authentication.ldap.useSSL=true
authentication.ldap.userObjectClass=posixAccount
authentication.ldap.usernameAttribute=uid

ambari-audit.log

2021-07-05T19:48:23.518+0200, User(null), RemoteIp(xxx.xxx.xxx.xxx), Operation(User login), Roles(
), Status(Failed), Reason(Authentication required)

ambari-server.log

02 Jul 2021 18:43:52,514  INFO [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:109 - Translated knox/<knox_gateway>@<REALM> to knox using auth-to-local rules during Kerberos authentication.
02 Jul 2021 18:43:52,515  WARN [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:143 - Failed find user account for user with username of knox during Kerberos authentication.
02 Jul 2021 18:43:52,516  WARN [ambari-client-thread-792188] AmbariKerberosAuthenticationFilter:149 - Negotiate Header was invalid: Negotiate YIIDlAYGKwYBBQUCoIIDiDCCA4SgDTALBgkqhkiG9xIBAgKhBAMCAXaiggNrBIIDZ2CCA2MGCSqGSIb3EgECAgEAboIDUjCCA06gAwIBBaEDAgEOogcDBQAgAAAAo4IB/WGCAfkwggH1oAMCAQWhMxsxMjZGNURFMDEtNUU0MC00RDhBLTk4QkQtQTQzNTNCN0JGNUUzLkRBVEFMQUtFLk9WSKJPME2gAwIBAKFGMEQbBEhUVFAbPG92aC1lbm9kZTYuMjZmNWRlMDEtNWU0MC00ZDhhLTk4YmQtYTQzNTNiN2JmNWUzLmRhdGFsYWtlLm92aKOCAWYwggFioAMCARKhAwIBAqKCAVQEggFQbqKii/FFb+dRh+NYor9rVsacj0GwwyOV1KCZLEFkyM7VEhjK/wV8gkJOL9V0u99lkAIPGFCI6TzHenX6VtiZEd/MvKox5b4V0REwnpTuX5vS5lXCfXtbEOF2SbXmch1/U5RCjVQr8+vm/8AYmILIsqnFFmU9AgFFtEtZsH1NZpiSbosJs3oDJ1yINbzuzrxpCJJTu2Rcv2j34mN8+SkA42D6ZO6yP1iwlTkWji9VEDSOYLdzzhPMq0rmUCuuHfDYHX2Dx09xx4h9C6d28E3+o07LmUTUOBbnkiwnk57HZ+muxfD7T93PHOcfcJGPPKFWnUzTBg/ZrNsU3M36Y6UdqJ3kH/nvu6TNZDy+EbJhztlz3H1xNlMQ7D9np+HNHd72CvgFFzEbLCqkR9WUUL0R9WIno10V2wX8gBXPV31qY+WU59vQHfa9kkmytR8a60sYpIIBNjCCATKgAwIBEqKCASkEggEle7R2DYTiWwLkpH4BKwKGmoZfD7ZX0garqKPDQ7HDBDJIXUTce4UELJ84bTwn/DtpnUeQ0rZ70FedqHlHutB60NsswXvzoozLk+7Meeo3c310iWNIEd++dtEo0ZUc4sE8CA984VPYHHral67L/L4rQgb39ikYhxo9ieLMMHZ7Im6uzsnQzH0shKSHgWsroneqSvQXAuNM+08OAWuP0znVFAddSpPiQYlnsf9LJcjrn0hLFQwoFV48HA7/PyGDZ8Jotw/UMD/sbokOazOvzubvGdoyS6+8xJlJnf+D9WneoviqOFn4Po6B8WANQuF02QbPOPeE4XKXA4n6iruZmdKeR1ksx6OuEGygRSs1lmsgOzZvrjNLCay2ty4qtsMUUCjQz9V8c4A=
org.springframework.security.core.userdetails.UsernameNotFoundException: Failed find user account for user with username of knox during Kerberos authentication.
        at org.apache.ambari.server.security.authentication.kerberos.AmbariAuthToLocalUserDetailsService.createUser(AmbariAuthToLocalUserDetailsService.java:144)
        at org.apache.ambari.server.security.authentication.kerberos.AmbariAuthToLocalUserDetailsService.loadUserByUsername(AmbariAuthToLocalUserDetailsService.java:110)
        at org.springframework.security.kerberos.authentication.KerberosServiceAuthenticationProvider.authenticate(KerberosServiceAuthenticationProvider.java:66)
        at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:156)
        at org.springframework.security.kerberos.web.authentication.SpnegoAuthenticationProcessingFilter.doFilter(SpnegoAuthenticationProcessingFilter.java:145)
        at org.apache.ambari.server.security.authentication.kerberos.AmbariKerberosAuthenticationFilter.doFilter(AmbariKerberosAuthenticationFilter.java:167)
        at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:120)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212)
        at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201)
        at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:745)

[#4]

@Raamarany advise please ?

[#5 · Accepted solution]

This issue occurs when kerberos authentication is enabled

There is bug issue opened in ambari jira https://issues.apache.org/jira/browse/AMBARI-25127

To fix my problem, i just disabled kerberos authentication

authentication.kerberos.enabled=false