NiFi Node showing 2 nodes and not respecting node down fault tolerance.

Question

Hello Guys,

We have 3 node NiFi cluster and the NiFi version is 1.12.1

nifi-hatest-01
nifi-hatest-02
nifi-hatest-03

All the 3 nodes are in cluster, but not in UI.

In all the 3 NiFi nodes, we see this 2 nodes in cluster mode:

Now, as we see nifi-hatest-02 is the primary and cooridantor. And nifi-hatest-03 is not being shown here.

In nifi-hatest-01, we see nifi-hatest-01 is connecting to nifi-hatest-02

2022-01-19 16:02:09,840 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2022-01-19 16:02:09,820 and sent to nifi-hatest-02:9088 at 2022-01-19 16:02:09,840; send took 20 millis

In nifi-hatest-03, we see nifi-hatest-03 is also connecting to nifi-hatest-02

2022-01-19 16:02:09,840 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2022-01-19 16:02:09,820 and sent to nifi-hatest-02:9088 at 2022-01-19 16:02:09,840; send took 20 millis

In nifi-hatest-02, we see nifi-hatest-02 is also connecting to all the 3 nodes as its coordinator:

2022-01-19 16:04:52,573 INFO [Process Cluster Protocol Request-30] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 35f2ed1a-ca6f-4cc6-ab4a-6c0774fc9c6d (type=HEARTBEAT, length=2837 bytes) from nifi-hatest-03:9091 in 19 millis
2022-01-19 16:04:52,782 INFO [Process Cluster Protocol Request-31] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 73d477cd-a936-4541-a854-03712cbc5fe9 (type=HEARTBEAT, length=2836 bytes) from nifi-hatest-02:9091 in 18 millis
2022-01-19 16:04:52,782 INFO [Clustering Tasks Thread-3] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2022-01-19 16:04:52,762 and sent to nifi-hatest-02:9088 at 2022-01-19 16:04:52,782; send took 19 millis
2022-01-19 16:04:53,916 INFO [Process Cluster Protocol Request-32] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 4fbc39d9-5154-4841-b3be-ddde58b22cd2 (type=HEARTBEAT, length=2836 bytes) from nifi-hatest-01:9091 in 19 millis

No errors regarding cluster in either of nodes. But still see 2 nodes only in UI.

Also Cluster is using zk as cluster state provider.

Another thing is that when we create a processor in any of the NiFi UI, it gets propagated to other nodes too! So functionality its kind of working.

DigitalPlumber · Accepted Answer

Reading your sample log messages closer I can see that the coordinator received a heartbeat from node 3

"2022-01-19 16:04:52,573 INFO [Process Cluster Protocol Request-30] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 35f2ed1a-ca6f-4cc6-ab4a-6c0774fc9c6d (type=HEARTBEAT, length=2837 bytes) from nifi-hatest-03:9091 in 19 millis"

So I also wonder if we have a caching rendering issue, can you see what the UI shows using incognito mode?

And finally if this is a DEV environment you can also try and delete your local state directory, that value is set in file "state-management.xml"

Deleting state will clear out any local state your processors might depend on if configured as such, so remove with caution.

It will also clear out cluster node ID's it local knows of.

DigitalPlumber · Answer

This to me sounds like a hostname issue.

could you confirm the value on nifi.properites for:

nifi.cluster.node.address=

nifi.web.https.host=

Those values should match for the name of the host.

If nothing there stands out check log entries for latest messages contain below strings:

"org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator: Status"

"org.apache.nifi.controller.StandardFlowService: Setting Flow Controller's Node ID:"

"org.apache.nifi.web.server.HostHeaderHandler"

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.