Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi cluster authorization error

avatar
Contributor

I used nifi 1.13.2 cluster of 2 nodes.
Node1 is rhel7, node2 is redos, embedded Zookeeper.
The cluster works correctly (nodes see each other, the coordinator is selected.)
Certificates on nodes are self-signed.

Faced with the problem of authorization in the UI.

LDAP authorization does not work (without cluster the LDAP authorization worked on both nodes).

I am trying to connect to node1.
When I enter my username and password, I return back to the authorization page.
The log on the node1:

2022-09-19 05:36:14,892 INFO [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://node1.domain.ru:8080/nifi-api/flow/current-user (source ip: <IP node1>)
2022-09-19 05:36:14,895 INFO [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for <user name>
2022-09-19 05:36:14,895 DEBUG [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: <user name>
2022-09-19 05:36:14,895 DEBUG [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: <user name>
2022-09-19 05:36:14,895 DEBUG [NiFi Web Server-258] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: <user name>
2022-09-19 05:36:14,901 DEBUG [NiFi Web Server-276] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-276] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <<user name>>
2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-276] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-09-19 05:36:14,902 INFO [NiFi Web Server-276] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<<user name>><CN=node1.domain.ru, OU=NIFI>) GET https://node1.domain.ru:8080/nifi-api/flow/current-user (source ip: <IP node1>)
2022-09-19 05:36:14,902 INFO [NiFi Web Server-276] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for <user name>
2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-276] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: <user name>
2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-276] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: <user name>
2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-276] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: <user name>
2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-276] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: <user name>
2022-09-19 05:36:15,228 DEBUG [NiFi Web Server-248] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.

 

The log on the node2:

2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-198] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-198] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-198] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-198] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-198] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-19 05:36:14,902 DEBUG [NiFi Web Server-198] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-19 05:36:14,902 INFO [NiFi Web Server-198] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<anonymous>) GET https://node2.domain.ru:8080/nifi-api/flow/current-user (source ip: <IP node1>)
2022-09-19 05:36:14,903 WARN [NiFi Web Server-198] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Anonymous authentication has not been configured.
2022-09-19 05:36:14,903 DEBUG [NiFi Web Server-198] o.a.n.w.s.NiFiAuthenticationFilter
org.apache.nifi.web.security.InvalidAuthenticationException: Anonymous authentication has not been configured.
at org.apache.nifi.web.security.anonymous.NiFiAnonymousAuthenticationProvider.authenticate(NiFiAnonymousAuthenticationProvider.java:46)
at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:79)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:100)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:100)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:100)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:100)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:487)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:336)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:301)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XContentTypeOptionsFilter.doFilter(XContentTypeOptionsFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XSSProtectionFilter.doFilter(XSSProtectionFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:834)

 

I will be glad of any help.

1 ACCEPTED SOLUTION

avatar
Contributor

Hello everyone, the problem was solved by replacing java Oracle openjdk-11.0.1.13 with java Liberica jdk11.0.16

View solution in original post

13 REPLIES 13

avatar
Master Mentor

@skoleg 

Do you have a network load balancer in front of your NiFi?
The user authentication token issued is only good for the NiFi host from which it was issued.  So if node 1 returns a client bearer token following successful authentication, and the load-balancer then sends subsequent request to node2, node 2 will not be able to accept that bearer token and return user to the login page. When using a external load-balancer, it is important to make sure sticky sessions are configured so that all redirects after login continue to get sent to same NiFi node.
----------------

If a load-balancer is not in play here, verify the same configuration in both your node's nifi.properties (except hostnames and keystore files), login-identity-provider.xml, and authorizers.xml files.

@Sanchari 

NiFi FlowFiles reside in connection between NiFi component processors.  When a processor gets a thread to execute, it takes the highest priority FlowFile from an inbound connection queue and executes the processor code utilizing that FlowFiles metadata/attributes and content (if processor needs content).  The FlowFile is not transferred to a processors outbound connection(s) until execution is complete.

When NiFi is shutdown gracefully (meaning a user has initiated a shutdown), NiFi stops scheduling future component execution.  NiFi then gives existing executing threads a grace period to complete their thread execution.  At the end of that grace period, any still running threads are killed with the JVM.  Since FlowFiles do not transfer to an outbound connection until code execution has completed, and FlowFile that was owned by a thread at the time the thread was killed still remains on the inbound connection.  When NiFi is started again and the dataflows started, the file processing will start over when the processor executes again and executes against the highest priority FlowFile in the connection.

Above being said, NiFi will favor data duplication over data loss every time.  It is possible in a small window of time that processor executes and part of that execution is let's say to write a file to a remote server.  NiFi may for example ack the completion of that transfer to the remote system and NiFi JVM was killed before internally it received ack back from target server. So the FlowFile would end up being processed again resulting potentially data duplication on the target server.  These are rare race conditions, but possible.

A restart is nothing more than a standard shutdown followed by a start.  The same behavior exists in the shutdown process as described above when a restart is performed.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

avatar
Contributor

Hi, Matt, thanks for your reply.

I don't have a network load balancer in front of my nifi.

I get an error even when starting both nodes with the initial configuration

avatar
Contributor

The same configuration in both node's.

avatar
Master Mentor

@skoleg 
Something is not configured the same if you are getting different behavior out of each node.
Unfortunately, without seeing your configuration files (nifi.properties, login-identity-providers.xml, authorizers.xml, authorizations.xml, and users.xml) and app-logs/user-logs, it would be difficult to provide  additional suggestions on your setup.

Make sure your NiFI nodes are authorized to proxy user requests, but i'd expect you to get an exception in the UI if they were not already.

"Anonymous" happens with no client/user authentication was successful.

Thanks,

Matt

avatar
Contributor

The first time I log in after restarting the nodes, I get a message:

2022-09-21 05:24:07,127 DEBUG [NiFi Web Server-236] o.a.n.w.s.NiFiAuthenticationFilter
org.apache.nifi.web.security.InvalidAuthenticationException: Anonymous authentication has not been configured.

In the future, when I try to log in from node1, I return to the authorization interface
In the logs of node1, I see the following:

== Proxy Entity Chain ==
Identity: username , IDP Groups: []
Identity: CN=node2.domain, OU=NIFI , IDP Groups: []
Identity: CN=node1.domain, OU=NIFI , IDP Groups: []
============
2022-09-21 06:05:29,191 INFO [NiFi Web Server-242] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for username
2022-09-21 06:05:29,191 DEBUG [NiFi Web Server-242] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 06:05:29,191 DEBUG [NiFi Web Server-242] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 06:05:29,191 DEBUG [NiFi Web Server-242] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 06:05:29,191 DEBUG [NiFi Web Server-242] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 07:35:37,938 DEBUG [NiFi Web Server-295] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-21 07:35:37,938 DEBUG [NiFi Web Server-295] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <username>
2022-09-21 07:35:37,938 DEBUG [NiFi Web Server-295] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-09-21 07:35:37,938 INFO [NiFi Web Server-295] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<username><CN=node2.domain, OU=NIFI>) GET https://node1.domain:8080/nifi-api/flow/current-user (source ip: IP node2)
2022-09-21 07:35:37,938 TRACE [NiFi Web Server-295] o.a.n.w.s.x.X509AuthenticationProvider
== Proxy Entity Chain ==
Identity: username , IDP Groups: []
Identity: CN=node2.domain, OU=NIFI , IDP Groups: []
============
2022-09-21 07:35:37,939 INFO [NiFi Web Server-295] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for username
2022-09-21 07:35:37,939 DEBUG [NiFi Web Server-295] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 07:35:37,939 DEBUG [NiFi Web Server-295] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 07:35:37,939 DEBUG [NiFi Web Server-295] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 07:35:37,939 DEBUG [NiFi Web Server-295] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 07:35:37,954 DEBUG [NiFi Web Server-284] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-21 07:35:37,954 DEBUG [NiFi Web Server-284] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <username><CN=node2.domain, OU=NIFI>
2022-09-21 07:35:37,954 DEBUG [NiFi Web Server-284] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-09-21 07:35:37,954 INFO [NiFi Web Server-284] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<username><CN=node2.domain, OU=NIFI><CN=node1.domain, OU=NIFI>) GET https://node1.domain:8080/nifi-api/flow/current-user (source ip: IP node1)
2022-09-21 07:35:37,954 TRACE [NiFi Web Server-284] o.a.n.w.s.x.X509AuthenticationProvider


When I try to log in from node 2, I first get an error

on the browser screen:
javax.net.ssl.SSLPeerUnverifiedException: Hostname node1.domain not verified (no certificates)


In the logs of node2:
2022-09-21 06:15:58,864 WARN [Replicate Request Thread-1] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to node1.domain:8080 due to javax.net.ssl.SSLPeerUnverifiedException: Hostname node1.domain not verified (no certificates)
2022-09-21 06:15:58,864 WARN [Replicate Request Thread-1] o.a.n.c.c.h.r.ThreadPoolRequestReplicator
javax.net.ssl.SSLPeerUnverifiedException: Hostname node1.domain not verified (no certificates)

on subsequent authorization attempts from node 2, I return to the authorization interface and:

in the logs of node1, I see the following:

2022-09-21 08:06:41,481 DEBUG [NiFi Web Server-310] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-21 08:06:41,481 DEBUG [NiFi Web Server-310] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <username>
2022-09-21 08:06:41,481 DEBUG [NiFi Web Server-310] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-09-21 08:06:41,481 INFO [NiFi Web Server-310] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<username><CN=node2.domain, OU=NIFI>) GET https://node1.domain:8080/nifi-api/flow/current-user (source ip: IP node2)
2022-09-21 08:06:41,481 TRACE [NiFi Web Server-310] o.a.n.w.s.x.X509AuthenticationProvider
== Proxy Entity Chain ==
Identity: username , IDP Groups: []
Identity: CN=node2.domain, OU=NIFI , IDP Groups: []
============
2022-09-21 08:06:41,481 INFO [NiFi Web Server-310] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for username
2022-09-21 08:06:41,481 DEBUG [NiFi Web Server-310] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 08:06:41,481 DEBUG [NiFi Web Server-310] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 08:06:41,481 DEBUG [NiFi Web Server-310] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 08:06:41,481 DEBUG [NiFi Web Server-310] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 08:06:41,487 DEBUG [NiFi Web Server-311] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-21 08:06:41,487 DEBUG [NiFi Web Server-311] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntitiesChain - <username><CN=node2.domain, OU=NIFI>
2022-09-21 08:06:41,487 DEBUG [NiFi Web Server-311] o.a.n.w.s.x509.X509AuthenticationFilter Raw X-ProxiedEntityGroups - <>
2022-09-21 08:06:41,488 INFO [NiFi Web Server-311] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<username><CN=node2.domain, OU=NIFI><CN=node1.domain, OU=NIFI>) GET https://node1.domain:8080/nifi-api/flow/current-user (source ip: IP node1)
2022-09-21 08:06:41,488 TRACE [NiFi Web Server-311] o.a.n.w.s.x.X509AuthenticationProvider
== Proxy Entity Chain ==
Identity: username , IDP Groups: []
Identity: CN=node2.domain, OU=NIFI , IDP Groups: []
Identity: CN=node1.domain, OU=NIFI , IDP Groups: []
============
2022-09-21 08:06:41,488 INFO [NiFi Web Server-311] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for username
2022-09-21 08:06:41,488 DEBUG [NiFi Web Server-311] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 08:06:41,488 DEBUG [NiFi Web Server-311] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 08:06:41,488 DEBUG [NiFi Web Server-311] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 08:06:41,488 DEBUG [NiFi Web Server-311] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username

in the logs of node2, I see the following:

2022-09-21 08:06:40,026 TRACE [NiFi Web Server-183] o.a.nifi.web.security.jwt.JwtService Generating JWT for LoginAuthenticationToken for username issued by LdapProvider expiring at 21-09-2022 20:06:40.025 [1663780000025 ms, 43199999 ms remaining]
2022-09-21 08:06:40,090 DEBUG [NiFi Web Server-212] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2022-09-21 08:06:41,473 DEBUG [NiFi Web Server-235] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-21 08:06:41,473 DEBUG [NiFi Web Server-235] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2022-09-21 08:06:41,473 DEBUG [NiFi Web Server-235] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-21 08:06:41,473 INFO [NiFi Web Server-235] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://node2.domain:8080/nifi-api/flow/current-user (source ip: IP user computer)
2022-09-21 08:06:41,475 INFO [NiFi Web Server-235] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for username
2022-09-21 08:06:41,475 DEBUG [NiFi Web Server-235] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 08:06:41,475 DEBUG [NiFi Web Server-235] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 08:06:41,475 DEBUG [NiFi Web Server-235] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: username
2022-09-21 08:06:41,489 DEBUG [NiFi Web Server-214] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-21 08:06:41,489 DEBUG [NiFi Web Server-214] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.
2022-09-21 08:06:41,489 DEBUG [NiFi Web Server-214] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-21 08:06:41,489 DEBUG [NiFi Web Server-214] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-21 08:06:41,489 DEBUG [NiFi Web Server-214] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-21 08:06:41,489 DEBUG [NiFi Web Server-214] o.a.n.w.s.NiFiAuthenticationFilter Checking secure context token: null
2022-09-21 08:06:41,489 INFO [NiFi Web Server-214] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<anonymous>) GET https://node2.domain:8080/nifi-api/flow/current-user (source ip: IP node1)
2022-09-21 08:06:41,490 WARN [NiFi Web Server-214] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Anonymous authentication has not been configured.
2022-09-21 08:06:41,490 DEBUG [NiFi Web Server-214] o.a.n.w.s.NiFiAuthenticationFilter
org.apache.nifi.web.security.InvalidAuthenticationException: Anonymous authentication has not been configured.
at org.apache.nifi.web.security.anonymous.NiFiAnonymousAuthenticationProvider.authenticate(NiFiAnonymousAuthenticationProvider.java:46)
at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:79)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:100)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:100)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:100)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:100)
at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlets.DoSFilter.doFilterChain(DoSFilter.java:487)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:336)
at org.eclipse.jetty.servlets.DoSFilter.doFilter(DoSFilter.java:301)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XContentTypeOptionsFilter.doFilter(XContentTypeOptionsFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XSSProtectionFilter.doFilter(XSSProtectionFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.nifi.web.security.headers.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-09-21 08:06:41,808 DEBUG [NiFi Web Server-45] o.a.n.w.s.x509.X509CertificateExtractor No client certificate found in request.


Please tell me how it is configured authorization of nodes for user proxy requests?

avatar
Contributor

At the moment, I have done the following:
1. Created the initial configuration on the first node.
2. Copied the configuration to the second node.
3. Launched both nodes.
4. Tried to log in on the first node.
5 . I got this screen.

skoleg_1-1663843996263.png

 


How can I insert configuration files and logs?

 

avatar
Master Mentor

@skoleg 
Looks like you may have an issue with your self signed node certificates.
Can you share the output of your keystore and truststore from both nodes:

keytool -v -list -keystore <keystore filename>
keytool -v -list -keystore <truststore filename>

  
I wonder if perhaps you are missing the required clientAuth ExtendedKeyUsage (EKU).

Thanks,

Matt

avatar
Contributor

@MattWho 

TRUSTSTORE.JKS
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 3 entries
Alias name: nifi-cert
Entry type: trustedCertEntry
Owner: CN=node1, OU=NIFI
Issuer: CN=node1, OU=NIFI
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [ key
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
Key_CertSign
Crl_Sign
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: node1
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [key
]
]


Alias name: node2
Creation date: Sep 22, 2022
Entry type: trustedCertEntry

Owner: CN=node2, OU=NIFI
Issuer: CN=node2, OU=NIFI
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [key
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
Key_CertSign
Crl_Sign
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: node2
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [key
]
]

 

KEYSTORE.JKS (NODE1)

Alias name: nifi-key
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=node1, OU=NIFI
Issuer: CN=node1, OU=NIFI
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [key
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: node1
DNSName: node1
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [key
]
]

Certificate[2]:
Owner: CN=node1, OU=NIFI
Issuer: CN=snode1, OU=NIFI
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [key

]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
Key_CertSign
Crl_Sign
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: node1
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [key
]
]

KEYSTORE.JKS (NODE2)

Alias name: nifi-key
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=node2, OU=NIFI
Issuer: CN=node2, OU=NIFI
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [key
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:false
PathLen: undefined
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: node2
DNSName: node2
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [key
]
]

Certificate[2]:
Owner: CN=node2, OU=NIFI
Issuer: CN=snode2, OU=NIFI
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [key

]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
Key_Agreement
Key_CertSign
Crl_Sign
]

#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: node2
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [key
]
]

 

I created certificates with this commands:

/nifi/nifi-toolkit-1.16.3/bin/tls-toolkit.sh standalone -c node1 -n node1 --days 3650 --keyStorePassword Passwd --trustStorePassword Passwd -o /nifi/CA/node1 --subjectAlternativeNames node1

 

/nifi/nifi-toolkit-1.16.3/bin/tls-toolkit.sh standalone -c node2 -n node2 --days 3650 --keyStorePassword Passwd --trustStorePassword Passwd -o /nifi/CA/node2 --subjectAlternativeNames node2

 

And created a single truststore.jks

keytool -import -alias node2 -file /nifi/CA//nifi-cert.pem -keystore /nifi/CA/truststore.jks

avatar
Contributor

Now I tried to log in to the second node and got an error:

 

2022-09-22 15:02:41,439 WARN [Replicate Request Thread-2] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to node1:8080 due to javax.net.ssl.SSLPeerUnverifiedException: Hostname node1 not verified (no certificates)
2022-09-22 15:02:41,439 WARN [Replicate Request Thread-2] o.a.n.c.c.h.r.ThreadPoolRequestReplicator
javax.net.ssl.SSLPeerUnverifiedException: Hostname node1 not verified (no certificates)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:396)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:132)
at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:126)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:652)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:844)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)