Unable to autheticate to NIFI API using loadbalancer

[#1]

We have an AWS loadbalancer setup for NIFI cluster.

Authentication works fine when accessing the NIFI UI using the loadbalancer url.

While trying to configure Site-to-Site, authorization errors are encountered.

On checking the logs, seems like the NIFI API is trying to authenticate loadbalancer using the user "anonymous" which doesnot exist.

Nifi UI AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi

Nifi API AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api

Loadbalancer : HTTPS Listener on Port 8443

Why NIFI is trying to access the api url "https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api" using "anonymous" user

Snippet : nifi-app.log

2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api due to org.apache.nifi.remote.util.SiteToSiteRestApiClient$HttpGetFailedException: response code 401:Unauthorized with explanation: null
2018-07-11 05:26:44,822 DEBUG [Timer-Driven Process Thread-7] o.a.n.r.util.SiteToSiteRestApiClient
org.apache.nifi.remote.util.SiteToSiteRestApiClient$HttpGetFailedException: response code 401:Unauthorized with explanation: null
        at org.apache.nifi.remote.util.SiteToSiteRestApiClient.execute(SiteToSiteRestApiClient.java:1145)
        at org.apache.nifi.remote.util.SiteToSiteRestApiClient.execute(SiteToSiteRestApiClient.java:1179)
        at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:374)
        at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:355)
        at org.apache.nifi.remote.util.SiteToSiteRestApiClient.getController(SiteToSiteRestApiClient.java:340)
        at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:796)
        at org.apache.nifi.controller.FlowController.updateRemoteProcessGroups(FlowController.java:4383)
        at org.apache.nifi.controller.FlowController.access$100(FlowController.java:254)
        at org.apache.nifi.controller.FlowController$3.run(FlowController.java:744)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController Unable to communicate with remote instance RemoteProcessGroup[https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi] due to org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi due to: response code 401:Unauthorized with explanation: null
2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController
org.apache.nifi.controller.exception.CommunicationsException: Unable to communicate with Remote NiFi at URI https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi due to: response code 401:Unauthorized with explanation: null
        at org.apache.nifi.remote.StandardRemoteProcessGroup.refreshFlowContents(StandardRemoteProcessGroup.java:817)
        at org.apache.nifi.controller.FlowController.updateRemoteProcessGroups(FlowController.java:4383)
        at org.apache.nifi.controller.FlowController.access$100(FlowController.java:254)
        at org.apache.nifi.controller.FlowController$3.run(FlowController.java:744)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Snippet : nifi-user.log

2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2018-07-11 05:34:44,968 DEBUG [NiFi Web Server-24] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2018-07-11 05:34:44,971 DEBUG [NiFi Web Server-24] o.a.n.w.s.a.NiFiAnonymousUserFilter Populated SecurityContextHolder with anonymous token: 'anonymous'
2018-07-11 05:34:44,971 INFO [NiFi Web Server-24] o.a.n.w.a.config.NotFoundExceptionMapper com.sun.jersey.api.NotFoundException: null for uri: https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api/controller/users. Returning Not Found response.
2018-07-11 05:34:44,972 DEBUG [NiFi Web Server-24] o.a.n.w.a.config.NotFoundExceptionMapper
com.sun.jersey.api.NotFoundException: null for uri: https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api/controller/users
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1543)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
        at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:122)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:83)
        at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:57)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:83)
        at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:57)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:83)
        at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:57)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
        at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1613)
        at org.apache.nifi.web.server.JettyServer$2.doFilter(JettyServer.java:908)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:541)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1593)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1239)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1562)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1141)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:118)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:561)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:564)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
        at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
        at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122)
        at org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:748)

[#2 · Accepted solution]

@Nikhil

NiFi Site-To-Site uses two-way TLS authentication.

-

Check to make sure the keystore file being used on each of your NiFi nodes contains a single "PrivateKeyEntry" and make sure the PrivateKeyEntry supports both the ClientAuth and ServerAuth key usage.

-

If the PrivateKeyEntry supports serverAuth only, the NiFi service will not be able to provide a client certificate in the TLS handshake.

-

I also noticed timestamps for entries in your nifi-user.log to not match with timestamps from the shared nifi-app.log file. The entries specifically shared are not directly related to one another.

-

Thank you,

Matt

-

[#3]

@Nikhil

-

*** Forum tip: Please try to avoid responding to an Answer by starting a new answer. Instead use the "add comment" tp respond to en existing answer. There is no guaranteed order to different answers which can make following a response thread difficult especially when multiple people are trying to assist you.

-

You get a verbose output form your keystore using the keytool command

-

keytool -v -list -keystore <keystore.jks file>

-

Look to see if your PrivateKeyEntry has any "ExtendedKeyUsages" listed.
It would look something like this:

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  clientAuth
  serverAuth
]

-

Since you commented that the RPG works correctly when you use the URLs for the nodes directly, the certificates must support clientAuth then. This sounds more like a LB configuration issue. The certificate is being sent to the LB, but the LB is not forwarding that client cert on to the target end-point.

-

It is also not clear to me why you would configure your RPG to point at your LB instead of at one or more of the NiFi nodes directly?
----- The RPG will retrieve details about the entire target NiFi cluster when it connects and store/update that locally. So there really is no need for a LB in front of the RPG.

-

Thank you,

Matt

[#4]

@Matt Clarke

Thanks for the info regarding the keystore.

Regarding

"Since you commented that the RPG works correctly when you use the URLs for the nodes directly, the certificates must support clientAuth then. This sounds more like a LB configuration issue. The certificate is being sent to the LB, but the LB is not forwarding that client cert on to the target end-point."

>> I am able to access NIFI UI using the LB url. If the loadbalancer is not working I should not get the UI as well. But here the issue is related to nifi-api access, from the servers, using LB url. But its still strange that I am able to access the UI. I believe that the web UI also uses API to get the access tokens , flow details and other details.

"It is also not clear to me why you would configure your RPG to point at your LB instead of at one or more of the NiFi nodes directly?"

>> Initially we were using a single NIFI instance as RPG but it was SPOF. So we thought of adding an LB on top of NIFI. If we add the list of NIFI url's , it would be difficult to update the RPG url list in scenarios like adding/removing nifi instance. Also RPG cannot be edited and it has to be recreated. In our case we have large number of workflows, so recreating them wont be a practical approach. Hope you got my point 🙂

Thank You

Nikhil

[#5]

@Nikhil

What I was getting at was that the authentication methods are different here.

-
I am assuming your users who access the NIFi UI via the load balancer are using a user/password authentication method? That method results in a token being issued to the authenticated user which is then passed by the client in every subsequent request to the NiFi API.

-

With Site-To-Site, there are no tokens involved in the authentication process since certificate authentication occurs via two-way TLS in every single rest api call.

-

Admittedly, I know nothing about your specific LB or how it is configured, so these are just suggested things to consider.

-

Also want to let you know you must be running an older HDF version. Newer versions support editing the URL string without needing to recreate the RPG.

-

Thank you,

Matt

[#6]

@Matt Clarke

Sorry for the delay in getting back.

Thanks for the info.

I was able to setup lb for nifi now.

Had to configure two lbs. ALB for webui and NLB for rpg.

Thank You

Nikhil

[#7]

Hi @nikhilr ,

I have set up a 3node secure nifi cluster. Now I am trying to set up AWS LB  for nifi web UI and getting an error saying [Anonymous authentication has not been configured.].Could you please help me how could I resolve the anonymous authentication issue while accessing from load balancer url.

o.a.n.w.s.NiFiAuthenticationFilter Authentication Failed 10.XXX.XXX.159 GET https://10.XXX.XXX.162:8443/nifi-api/flow/status [Anonymous authentication has not been configured.]

[#8]

@Naresh_n as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

[#9]

Hi Matt,

Thanks for your update and sorry for the delay in getting back

Regarding,

Check to make sure the keystore file being used on each of your NiFi nodes contains a single "PrivateKeyEntry" and make sure the PrivateKeyEntry supports both the ClientAuth and ServerAuth key usage.

If the PrivateKeyEntry supports serverAuth only, the NiFi service will not be able to provide a client certificate in the TLS handshake.

I am using a self signed certificate for all the NIFI servers and Load Balancer, which is signed by a private CA. Each NIFI certificates has the its hostname and LB name as SAN. All these certificates has only a single private key for each.

I have also used the toolkit for creating the SSL certificates for NIFI servers and LB. But still the results are the same.

Also if the PrivateKeyEntry doesnot support both ClientAuth and ServerAuth, it should not work if I provide a single NIFI server url or the group of NIFI server urls in the RPG. But in my case it works

Regarding

I also noticed timestamps for entries in your nifi-user.log to not match with timestamps from the shared nifi-app.log file. The entries specifically shared are not directly related to one another.

You can ignore the timestamps. There are sync issues. I copied it randomly.

Also i have a query regarding "If the PrivateKeyEntry supports serverAuth only, the NiFi service will not be able to provide a client certificate in the TLS handshake." >> Is there a way to find out whether the private key supports both client and server auth ?

Thank You

Nikhil