Architecting AWS Load Balancers for Apache NiFi Mutual TLS and Site-to-Site Traffic
We have an AWS loadbalancer setup for NIFI cluster.
Authentication works fine when accessing the NIFI UI using the loadbalancer url.
While trying to configure Site-to-Site, authorization errors are encountered.
On checking the logs, seems like the NIFI API is trying to authenticate loadbalancer using the user "anonymous" which doesnot exist.
Nifi UI AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi
Nifi API AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api
Loadbalancer : HTTPS Listener on Port 8443
Why NIFI is trying to access the api url "https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api" using "anonymous" user
Snippet : nifi-app.log
2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api due to org.apache.nifi.remote.util.SiteToSiteRestApiClient$HttpGetFailedException: response code 401:Unauthorized with explanation: null
2018-07-11 05:26:44,822 DEBUG [Timer-Driven Process Thread-7] o.a.n.r.util.SiteToSiteRestApiClient
org.apache.nifi.remote.util.SiteToSiteRestApiClient$HttpGetFailedException: response code 401:Unauthorized with explanation: null
at org.apache.nifi.remote.util.SiteToSiteRestApiClient.execute(SiteToSiteRestApiClient.java:1145)
at org.apache.nifi.remote.util.SiteToSiteRestApiClient.execute(SiteToSiteRestApiClient.java:1179)
at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:374)
at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:355)
at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:340)
at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:796)
at org.apache.nifi.controller.FlowController.updateRemoteProcessGroups(FlowController.java:4383)
at org.apache.nifi.controller.FlowController.access$100(FlowController.java:254)
at org.apache.nifi.controller.FlowController$3.run(FlowController.java:744)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController Unable to communicate with remote instance RemoteProcessGroup[https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi] due to org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi due to: response code 401:Unauthorized with explanation: null
2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController
org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi due to: response code 401:Unauthorized with explanation: null
at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:817)
at org.apache.nifi.controller.FlowController.updateRemoteProcessGroups(FlowController.java:4383)
at org.apache.nifi.controller.FlowController.access$100(FlowController.java:254)
at org.apache.nifi.controller.FlowController$3.run(FlowController.java:744)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Snippet : nifi-user.log
2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2018-07-11 05:34:44,971 DEBUG [NiFi Web Server-24] o.a.n.w.s.a.NiFiAnonymousUserFilter Populated SecurityContextHolder with anonymous token: 'anonymous'
2018-07-11 05:34:44,971 INFO [NiFi Web Server-24] o.a.n.w.a.config.NotFoundExceptionMapper com.sun.jersey.api.NotFoundException: null for uri: https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api/controller/users. Returning Not Found response.
2018-07-11 05:34:44,972 DEBUG [NiFi Web Server-24] o.a.n.w.a.config.NotFoundExceptionMapper
com.sun.jersey.api.NotFoundException: null for uri: https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api/controller/users
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1543)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:122)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:83)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:57)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:83)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:57)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:83)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:57)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1613)
at org.apache.nifi.web.server.JettyServer$2.doFilter(JettyServer.java:908)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:541)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1593)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1239)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1562)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1141)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:118)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:561)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122)
at org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:748)

