Created 07-22-2021 02:58 AM
Our ambari webui is encountering some errors when ambari-server restart. When some users are connected and we restart the ambari-server, it doesn't logout connected users and 401 errors appears
This behaviour is very constraining because it generate lot of lines in ambari logs (Thread)
How to force ambari to log out all connected users when ambari-server restart ?
Created 08-12-2021 07:54 AM
This issue occurs when kerberos authentication is enabled
There is bug issue opened in ambari jira https://issues.apache.org/jira/browse/AMBARI-25127
To fix my problem, i just disabled kerberos authentication
authentication.kerberos.enabled=false
Created 07-31-2021 02:42 AM
- Can you confirm the value you have set for user.inactivity.timeout.default, user.inactivity.timeout.role.readonly.default
- Can you share the complete 401 error users are experiencing?
- Can you confirm if you have spnego authentication enabled for ambari?
- Attach the below logs after 401 error occurs in ambari UI
1. ambari.properties file 2. ambari-audit.log
3. ambari-server.log
Created on 08-02-2021 07:57 AM - edited 08-02-2021 07:58 AM
Yes, I'm using spnego authentication with user inactivity properties
user.inactivity.timeout.default=600
user.inactivity.timeout.role.readonly.default=300
My ambari is behind a loadbalancer (nginx), bellow the 401 error logs :
/var/log/nginx/access.log
"GET /gateway/default/ambari/api/v1/clusters/prod/requests?to=end&page_size=10&fields=Requests&_=1625812254413 HTTP/1.1" 401 51 "https://knox.26f5de01-5e40-4d8a-98bd-a4353b7bf5e3.datalake/gateway/default/ambari" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0" "-"
ambari.properties
agent.package.install.task.timeout=36000
agent.stack.retry.on_repo_unavailability=false
agent.stack.retry.tries=5
agent.task.timeout=2000
agent.threadpool.size.max=25
ambari-server.user=root
ambari.ldap.isConfigured=true
ambari.post.user.creation.hook=/var/lib/ambari-server/resources/scripts/post-user-creation-hook.sh
ambari.post.user.creation.hook.enabled=true
ambari.python.wrap=ambari-python-wrap
authentication.kerberos.auth_to_local.rules=DEFAULT
authentication.kerberos.enabled=true
authentication.kerberos.spnego.keytab.file=/etc/security/keytabs/spnego.service.keytab
authentication.kerberos.spnego.principal=HTTP/<ambari_host_fqdn>
authentication.kerberos.user.types=LDAP
authentication.ldap.baseDn=cn=accounts,dc=<domain>,dc=<domain>,dc=<domain>
authentication.ldap.bindAnonymously=false
authentication.ldap.dnAttribute=dn
authentication.ldap.groupMembershipAttr=member
authentication.ldap.groupNamingAttr=cn
authentication.ldap.groupObjectClass=posixGroup
authentication.ldap.managerDn=uid=ldapbind,cn=sysaccounts,cn=etc,dc=<domain>,dc=<domain>,dc=<domain>
authentication.ldap.managerPassword=/etc/ambari-server/conf/ldap-password.dat
authentication.ldap.primaryUrl=<ipa_host_fqdn>:636
authentication.ldap.useSSL=true
authentication.ldap.userObjectClass=posixAccount
authentication.ldap.usernameAttribute=uid
ambari-audit.log
2021-07-05T19:48:23.518+0200, User(null), RemoteIp(xxx.xxx.xxx.xxx), Operation(User login), Roles(
), Status(Failed), Reason(Authentication required)
ambari-server.log
02 Jul 2021 18:43:52,514 INFO [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:109 - Translated knox/<knox_gateway>@<REALM> to knox using auth-to-local rules during Kerberos authentication.
02 Jul 2021 18:43:52,515 WARN [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:143 - Failed find user account for user with username of knox during Kerberos authentication.
02 Jul 2021 18:43:52,516 WARN [ambari-client-thread-792188] AmbariKerberosAuthenticationFilter:149 - Negotiate Header was invalid: Negotiate YIIDlAYGKwYBBQUCoIIDiDCCA4SgDTALBgkqhkiG9xIBAgKhBAMCAXaiggNrBIIDZ2CCA2MGCSqGSIb3EgECAgEAboIDUjCCA06gAwIBBaEDAgEOogcDBQAgAAAAo4IB/WGCAfkwggH1oAMCAQWhMxsxMjZGNURFMDEtNUU0MC00RDhBLTk4QkQtQTQzNTNCN0JGNUUzLkRBVEFMQUtFLk9WSKJPME2gAwIBAKFGMEQbBEhUVFAbPG92aC1lbm9kZTYuMjZmNWRlMDEtNWU0MC00ZDhhLTk4YmQtYTQzNTNiN2JmNWUzLmRhdGFsYWtlLm92aKOCAWYwggFioAMCARKhAwIBAqKCAVQEggFQbqKii/FFb+dRh+NYor9rVsacj0GwwyOV1KCZLEFkyM7VEhjK/wV8gkJOL9V0u99lkAIPGFCI6TzHenX6VtiZEd/MvKox5b4V0REwnpTuX5vS5lXCfXtbEOF2SbXmch1/U5RCjVQr8+vm/8AYmILIsqnFFmU9AgFFtEtZsH1NZpiSbosJs3oDJ1yINbzuzrxpCJJTu2Rcv2j34mN8+SkA42D6ZO6yP1iwlTkWji9VEDSOYLdzzhPMq0rmUCuuHfDYHX2Dx09xx4h9C6d28E3+o07LmUTUOBbnkiwnk57HZ+muxfD7T93PHOcfcJGPPKFWnUzTBg/ZrNsU3M36Y6UdqJ3kH/nvu6TNZDy+EbJhztlz3H1xNlMQ7D9np+HNHd72CvgFFzEbLCqkR9WUUL0R9WIno10V2wX8gBXPV31qY+WU59vQHfa9kkmytR8a60sYpIIBNjCCATKgAwIBEqKCASkEggEle7R2DYTiWwLkpH4BKwKGmoZfD7ZX0garqKPDQ7HDBDJIXUTce4UELJ84bTwn/DtpnUeQ0rZ70FedqHlHutB60NsswXvzoozLk+7Meeo3c310iWNIEd++dtEo0ZUc4sE8CA984VPYHHral67L/L4rQgb39ikYhxo9ieLMMHZ7Im6uzsnQzH0shKSHgWsroneqSvQXAuNM+08OAWuP0znVFAddSpPiQYlnsf9LJcjrn0hLFQwoFV48HA7/PyGDZ8Jotw/UMD/sbokOazOvzubvGdoyS6+8xJlJnf+D9WneoviqOFn4Po6B8WANQuF02QbPOPeE4XKXA4n6iruZmdKeR1ksx6OuEGygRSs1lmsgOzZvrjNLCay2ty4qtsMUUCjQz9V8c4A=
org.springframework.security.core.userdetails.UsernameNotFoundException: Failed find user account for user with username of knox during Kerberos authentication.
at org.apache.ambari.server.security.authentication.kerberos.AmbariAuthToLocalUserDetailsService.createUser(AmbariAuthToLocalUserDetailsService.java:144)
at org.apache.ambari.server.security.authentication.kerberos.AmbariAuthToLocalUserDetailsService.loadUserByUsername(AmbariAuthToLocalUserDetailsService.java:110)
at org.springframework.security.kerberos.authentication.KerberosServiceAuthenticationProvider.authenticate(KerberosServiceAuthenticationProvider.java:66)
at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:156)
at org.springframework.security.kerberos.web.authentication.SpnegoAuthenticationProcessingFilter.doFilter(SpnegoAuthenticationProcessingFilter.java:145)
at org.apache.ambari.server.security.authentication.kerberos.AmbariKerberosAuthenticationFilter.doFilter(AmbariKerberosAuthenticationFilter.java:167)
at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:120)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212)
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201)
at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Created 08-12-2021 01:50 AM
@Raamarany advise please ?
Created 08-12-2021 07:54 AM
This issue occurs when kerberos authentication is enabled
There is bug issue opened in ambari jira https://issues.apache.org/jira/browse/AMBARI-25127
To fix my problem, i just disabled kerberos authentication
authentication.kerberos.enabled=false