Support Questions

enirys · ‎07-02-2021

recently ambari server logs are showing some warnings

/var/log/ambari-server/ambari-server.log

02 Jul 2021 18:43:52,514  INFO [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:109 - Translated knox/<knox_gateway>@<REALM> to knox using auth-to-local rules during Kerberos authentication.
02 Jul 2021 18:43:52,515  WARN [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:143 - Failed find user account for user with username of knox during Kerberos authentication.
02 Jul 2021 18:43:52,516  WARN [ambari-client-thread-792188] AmbariKerberosAuthenticationFilter:149 - Negotiate Header was invalid: Negotiate YIIDlAYGKwYBBQUCoIIDiDCCA4SgDTALBgkqhkiG9xIBAgKhBAMCAXaiggNrBIIDZ2CCA2MGCSqGSIb3EgECAgEAboIDUjCCA06gAwIBBaEDAgEOogcDBQAgAAAAo4IB/WGCAfkwggH1oAMCAQWhMxsxMjZGNURFMDEtNUU0MC00RDhBLTk4QkQtQTQzNTNCN0JGNUUzLkRBVEFMQUtFLk9WSKJPME2gAwIBAKFGMEQbBEhUVFAbPG92aC1lbm9kZTYuMjZmNWRlMDEtNWU0MC00ZDhhLTk4YmQtYTQzNTNiN2JmNWUzLmRhdGFsYWtlLm92aKOCAWYwggFioAMCARKhAwIBAqKCAVQEggFQbqKii/FFb+dRh+NYor9rVsacj0GwwyOV1KCZLEFkyM7VEhjK/wV8gkJOL9V0u99lkAIPGFCI6TzHenX6VtiZEd/MvKox5b4V0REwnpTuX5vS5lXCfXtbEOF2SbXmch1/U5RCjVQr8+vm/8AYmILIsqnFFmU9AgFFtEtZsH1NZpiSbosJs3oDJ1yINbzuzrxpCJJTu2Rcv2j34mN8+SkA42D6ZO6yP1iwlTkWji9VEDSOYLdzzhPMq0rmUCuuHfDYHX2Dx09xx4h9C6d28E3+o07LmUTUOBbnkiwnk57HZ+muxfD7T93PHOcfcJGPPKFWnUzTBg/ZrNsU3M36Y6UdqJ3kH/nvu6TNZDy+EbJhztlz3H1xNlMQ7D9np+HNHd72CvgFFzEbLCqkR9WUUL0R9WIno10V2wX8gBXPV31qY+WU59vQHfa9kkmytR8a60sYpIIBNjCCATKgAwIBEqKCASkEggEle7R2DYTiWwLkpH4BKwKGmoZfD7ZX0garqKPDQ7HDBDJIXUTce4UELJ84bTwn/DtpnUeQ0rZ70FedqHlHutB60NsswXvzoozLk+7Meeo3c310iWNIEd++dtEo0ZUc4sE8CA984VPYHHral67L/L4rQgb39ikYhxo9ieLMMHZ7Im6uzsnQzH0shKSHgWsroneqSvQXAuNM+08OAWuP0znVFAddSpPiQYlnsf9LJcjrn0hLFQwoFV48HA7/PyGDZ8Jotw/UMD/sbokOazOvzubvGdoyS6+8xJlJnf+D9WneoviqOFn4Po6B8WANQuF02QbPOPeE4XKXA4n6iruZmdKeR1ksx6OuEGygRSs1lmsgOzZvrjNLCay2ty4qtsMUUCjQz9V8c4A=
org.springframework.security.core.userdetails.UsernameNotFoundException: Failed find user account for user with username of knox during Kerberos authentication.
        at org.apache.ambari.server.security.authentication.kerberos.AmbariAuthToLocalUserDetailsService.createUser(AmbariAuthToLocalUserDetailsService.java:144)
        at org.apache.ambari.server.security.authentication.kerberos.AmbariAuthToLocalUserDetailsService.loadUserByUsername(AmbariAuthToLocalUserDetailsService.java:110)
        at org.springframework.security.kerberos.authentication.KerberosServiceAuthenticationProvider.authenticate(KerberosServiceAuthenticationProvider.java:66)
        at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:156)
        at org.springframework.security.kerberos.web.authentication.SpnegoAuthenticationProcessingFilter.doFilter(SpnegoAuthenticationProcessingFilter.java:145)
        at org.apache.ambari.server.security.authentication.kerberos.AmbariKerberosAuthenticationFilter.doFilter(AmbariKerberosAuthenticationFilter.java:167)
        at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:120)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212)
        at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201)
        at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:745)

/var/log/ambari-server/ambari-audit.log

2021-07-02T19:01:16.881+0200, User(null), RemoteIp(xxx.xxx.xxx.xxx), Operation(User login), Roles(
), Status(Failed), Reason(Failed find user account for user with username of knox during Kerberos authentication.)

There is no identified issue right now, but it generate huge logs (50 lines/s) and we are unable to read ambari-logs

Ambari version : 2.6.2.2

HDP Version : HDP-2.6.5.1100

enirys · ‎07-22-2021

I finaly found the root cause of this issue, it happens when users connect to ambari web ui and the stay connected for a while then the session is killed due to timeout.

In my configuration there are no timeout for :

user.inactivity.timeout.default, user.inactivity.timeout.role.readonly.default

Setting these properties allows ambari to logout users after a period of inactivity.

This error may occurs also when we restart the ambari-server, users still connected with a 401 error message

i'm closing this thread and opening a new one with more details about the issue

https://community.cloudera.com/t5/Support-Questions/Logout-ambari-s-connected-users-on-ambari-server...

View solution in original post

pvishnu · ‎07-05-2021

Hi @enirys ,

It appears that you have configured Ambari to authenticate using Kerberos tokens via SPNEGO.

From the logs, I see you are trying to authenticate to Ambari using the principal "knox/<knox_gateway>@<REALM>" which gets translated to the user name "knox" and then searches it in the internal database or from an external source, such as an LDAP directory based on your configuration and unable to find it and hence the below warning message,

----------

02 Jul 2021 18:43:52,515  WARN [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:143 - Failed find user account for user with username of knox during Kerberos authentication.

----------

So validate the values set for the property "authentication.kerberos.user.types", and ensure the "knox" user is present in the User type mentioned and create if unavailable. Once available and when you re-authenticate again, Ambari will be able to find the relevant user and it bypasses the default user name and password login facility and should be able to authenticate successfully.

Let us know if this helps!

Thanks,
Prashanth Vishnu

enirys · ‎07-05-2021

Hi @pvishnu

The connection to ambari is done over knox gateway, and we have many clients using the gateway so i don't know who is causing this trouble

We have the same configuration on a dev cluster without having this issue except knox and ambari are not on the same node

as it use knox to authenticate i have created a knox/<hostname> principal on ambari node but without success, just duplicate number of logs and a new error appears in ambari-audit.log

we are using freeipa server to manage users and kerbers

2021-07-05T19:48:23.518+0200, User(null), RemoteIp(xxx.xxx.xxx.xxx), Operation(User login), Roles(
), Status(Failed), Reason(Authentication required)

i searched the property authentication.kerberos.user.types but can't find it on any component

pvishnu · ‎07-06-2021

Hi @enirys,

If you wanted to use the Authentication provider used with the Knox gateway for the Ambari UI and don't prefer the kerberos authentication, then you can disable it by setting the below property to false,

authentication.kerberos.enabled = false

You can find the "authentication.kerberos.*" properties in the /etc/ambari-server/conf/ambari.properties file.

Restart Ambari server after making necessary changes and monitor the logs to see if the warnings are repeated again.

enirys · ‎07-06-2021

Hi @pvishnu

I wan't to keep kerberos authentication, below ambari config

agent.package.install.task.timeout=36000
agent.stack.retry.on_repo_unavailability=false
agent.stack.retry.tries=5
agent.task.timeout=2000
agent.threadpool.size.max=25
ambari-server.user=root
ambari.ldap.isConfigured=true
ambari.post.user.creation.hook=/var/lib/ambari-server/resources/scripts/post-user-creation-hook.sh
ambari.post.user.creation.hook.enabled=true
ambari.python.wrap=ambari-python-wrap
authentication.kerberos.auth_to_local.rules=DEFAULT
authentication.kerberos.enabled=true
authentication.kerberos.spnego.keytab.file=/etc/security/keytabs/spnego.service.keytab
authentication.kerberos.spnego.principal=HTTP/<ambari_host_fqdn>
authentication.kerberos.user.types=LDAP
authentication.ldap.baseDn=cn=accounts,dc=<domain>,dc=<domain>,dc=<domain>
authentication.ldap.bindAnonymously=false
authentication.ldap.dnAttribute=dn
authentication.ldap.groupMembershipAttr=member
authentication.ldap.groupNamingAttr=cn
authentication.ldap.groupObjectClass=posixGroup
authentication.ldap.managerDn=uid=ldapbind,cn=sysaccounts,cn=etc,dc=<domain>,dc=<domain>,dc=<domain>
authentication.ldap.managerPassword=/etc/ambari-server/conf/ldap-password.dat
authentication.ldap.primaryUrl=<ipa_host_fqdn>:636
authentication.ldap.useSSL=true
authentication.ldap.userObjectClass=posixAccount
authentication.ldap.usernameAttribute=uid

I just need to know the origin of previous warning and remove them, i suspect ambari views which are used by some users but i'm not sure.

enirys · ‎07-09-2021

@pvishnu

Any update please ? do you need more informations ?

pvishnu · ‎07-13-2021

@enirys , Try checking the Ambari audit logs to identify where the authentication requests using the Knox kerberos principal is originating from.

enirys · ‎07-22-2021