Member since
01-10-2018
22
Posts
1
Kudos Received
0
Solutions
07-18-2018
05:13 AM
@Matt Clarke Sorry for the delay in getting back. Thanks for the info. I was able to setup lb for nifi now. Had to configure two lbs. ALB for webui and NLB for rpg. Thank You Nikhil
... View more
07-13-2018
01:18 AM
@Matt Clarke Thanks for the info regarding the keystore. Regarding "Since you commented that the RPG works correctly when you use the URLs for the nodes directly, the certificates must support clientAuth then. This sounds more like a LB configuration issue. The certificate is being sent to the LB, but the LB is not forwarding that client cert on to the target end-point." >> I am able to access NIFI UI using the LB url. If the loadbalancer is not working I should not get the UI as well. But here the issue is related to nifi-api access, from the servers, using LB url. But its still strange that I am able to access the UI. I believe that the web UI also uses API to get the access tokens , flow details and other details. "It is also not clear to me why you would configure your RPG to point at your LB instead of at one or more of the NiFi nodes directly?" >> Initially we were using a single NIFI instance as RPG but it was SPOF. So we thought of adding an LB on top of NIFI. If we add the list of NIFI url's , it would be difficult to update the RPG url list in scenarios like adding/removing nifi instance. Also RPG cannot be edited and it has to be recreated. In our case we have large number of workflows, so recreating them wont be a practical approach. Hope you got my point 🙂 Thank You Nikhil
... View more
07-12-2018
11:59 AM
Hi Matt, Thanks for your update and sorry for the delay in getting back Regarding, Check to make sure the keystore file being used on each of your NiFi nodes contains a single "PrivateKeyEntry" and make sure the PrivateKeyEntry supports both the ClientAuth and ServerAuth key usage.
If the PrivateKeyEntry supports serverAuth only, the NiFi service will not be able to provide a client certificate in the TLS handshake.
I am using a self signed certificate for all the NIFI servers and Load Balancer, which is signed by a private CA. Each NIFI certificates has the its hostname and LB name as SAN. All these certificates has only a single private key for each. I have also used the toolkit for creating the SSL certificates for NIFI servers and LB. But still the results are the same. Also if the PrivateKeyEntry doesnot support both ClientAuth and ServerAuth, it should not work if I provide a single NIFI server url or the group of NIFI server urls in the RPG. But in my case it works Regarding I also noticed timestamps for entries in your nifi-user.log to not match with timestamps from the shared nifi-app.log file. The entries specifically shared are not directly related to one another. You can ignore the timestamps. There are sync issues. I copied it randomly. Also i have a query regarding "If the PrivateKeyEntry supports serverAuth only, the NiFi service will not be able to provide a client certificate in the TLS handshake." >> Is there a way to find out whether the private key supports both client and server auth ? Thank You Nikhil
... View more
07-11-2018
05:58 AM
We have an AWS loadbalancer setup for NIFI cluster.
Authentication works fine when accessing the NIFI UI using the loadbalancer url.
While trying to configure Site-to-Site, authorization errors are encountered.
On checking the logs, seems like the NIFI API is trying to authenticate loadbalancer using the user "anonymous" which doesnot exist.
Nifi UI AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi
Nifi API AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api
Loadbalancer : HTTPS Listener on Port 8443
Why NIFI is trying to access the api url "https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api" using "anonymous" user
Snippet : nifi-app.log
2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api due to org.apache.nifi.remote.util.SiteToSiteRestApiClient$HttpGetFailedException: response code 401:Unauthorized with explanation: null
2018-07-11 05:26:44,822 DEBUG [Timer-Driven Process Thread-7] o.a.n.r.util.SiteToSiteRestApiClient
org.apache.nifi.remote.util.SiteToSiteRestApiClient$HttpGetFailedException: response code 401:Unauthorized with explanation: null
at org.apache.nifi.remote.util.SiteToSiteRestApiClient.execute(SiteToSiteRestApiClient.java:1145)
at org.apache.nifi.remote.util.SiteToSiteRestApiClient.execute(SiteToSiteRestApiClient.java:1179)
at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:374)
at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:355)
at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:340)
at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:796)
at org.apache.nifi.controller.FlowController.updateRemoteProcessGroups(FlowController.java:4383)
at org.apache.nifi.controller.FlowController.access$100(FlowController.java:254)
at org.apache.nifi.controller.FlowController$3.run(FlowController.java:744)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController Unable to communicate with remote instance RemoteProcessGroup[https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi] due to org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi due to: response code 401:Unauthorized with explanation: null
2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController
org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi due to: response code 401:Unauthorized with explanation: null
at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:817)
at org.apache.nifi.controller.FlowController.updateRemoteProcessGroups(FlowController.java:4383)
at org.apache.nifi.controller.FlowController.access$100(FlowController.java:254)
at org.apache.nifi.controller.FlowController$3.run(FlowController.java:744)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Snippet : nifi-user.log
2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2018-07-11 05:34:44,971 DEBUG [NiFi Web Server-24] o.a.n.w.s.a.NiFiAnonymousUserFilter Populated SecurityContextHolder with anonymous token: 'anonymous'
2018-07-11 05:34:44,971 INFO [NiFi Web Server-24] o.a.n.w.a.config.NotFoundExceptionMapper com.sun.jersey.api.NotFoundException: null for uri: https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api/controller/users. Returning Not Found response.
2018-07-11 05:34:44,972 DEBUG [NiFi Web Server-24] o.a.n.w.a.config.NotFoundExceptionMapper
com.sun.jersey.api.NotFoundException: null for uri: https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api/controller/users
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1543)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:122)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:83)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:57)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:83)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:57)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:83)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:57)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1613)
at org.apache.nifi.web.server.JettyServer$2.doFilter(JettyServer.java:908)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:541)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1593)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1239)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1562)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1141)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:118)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:561)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122)
at org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:748)
... View more
Labels:
- Labels:
-
Apache NiFi