Failed to replicate request GET /nifi-api/flow/current-user to desktop:11443 java.net.ConnectException: Failed to connect to desktop/192.1.1.39:11443

[#1]

Hi,

I am testing nifi 2.0.0-M4 in cluster mode (secured mode).
But whenever I start the second node I get the following error when connecting to nifi UI.

024-09-07 13:50:16,976 WARN [Replicate Request Thread-107] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to desktop:11443
java.net.ConnectException: Failed to connect to desktop/192.1.1.39:11443
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:139)
at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:133)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:648)
at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:872)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:549)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:751)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
... 24 common frames omitted

Zookeeper is running separetly. and configs in nifi property look lie that:

nifi.remote.input.socket.port=10443
nifi.web.http.port=
nifi.web.https.port=11443
nifi.listener.bootstrap.port=0
nifi.cluster.node.protocol.port=12443
nifi.cluster.load.balance.port=6342
nifi.remote.input.host=<hostname>
nifi.web.http.host=
nifi.web.https.host=<hostname>
nifi.web.proxy.host=
nifi.cluster.load.balance.host=<hostname>
nifi.zookeeper.connect.string=<zookeeperhost>:2181

Also each hosts file of computer is properly set, thus IP address point the same hostnames specified in the file.
One computer is linux, the one in error is windows (firewall removed).
SSL cert has been generated with nifi-toolkit.
Everyone is reachable from a network point of view (zookeeper can ping the two nodes, and the two nodes can ping each other and also zookeeper).

I am clueless right now.

Thanks for your guidance

 

[#2]

@P2as 

Zookeeper needs to have more then 1 node in order to have quorum. A single ZK instance will not work.  

The next question is how is client/user authentication and authorization setup?
While i dod not see any untrusted proxy exception or SSL exceptions in what you shared, I wonder if you are encountering a mutualTLS issue between your two nodes resulting in your connection exception.

When you try to access a NiFi node's URL, your request is replicated by the elected cluster coordinator to all nodes in the cluster and those nodes need to respond with what access the authenticated user is authorized for on each node.  It is this replication request that is failing.

You may need to dig a bit deeper in to your logs and configurations making sure that the NiFi instances successfully bound to those ports and those ports are not being blocked.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Thank you @MattWho 
In fact I have the quorum as I have 3 zookeepers nodes (just one is exposed to the nifi instances).
For authorization and auth, it's a SingleUserAuthorizer implementation . Unfortunaly, I didn't find anything helpfull in the log.

If I shutdown one of the node (not the primary), I can access the UI, seeing 1/2 nodes in the cluster.

[#4]

I think you issue may be with using the SingleUserAuthorizer and Single user login provider.  These out of the box providers were built so that NiFi could be HTTPS enabled securely out of the box.  They are not designed to support clustering, nor are they suitable for production NiFi.

You'll want to configure your NiFi cluster to use a production ready authorizer (managed authorizer) and user authentication method other then single user so you can have granular access controls per user/team.  Most common is the ldap-provider.

The documentation provides examples for authorizer.xml setup:
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-based-ldap-authentication

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-based-kerberos-authentica...

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap-based-usersgroups-referen...

ETC....

You cluster is most likely not forming completely due to node to node authentication and authorization issue resulting from using the single user authorizer.   In a NiFi cluster the node identities (derived from clientAuth certificates in the Mutual TLS exchange) need to be authorized against some NiFi policies like "Proxy user requests".

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt