Nodes Not Joining the NiFi Cluster

[#1]

Following the Apache walkthrough titled "NiFi Cluster Using NiFi CA", i created a 3 node cluster using 3 separate aws instances.  However, when I access the GUI at https://nifi1:9337 (or nifi2, or nifi3), for nifi1 and nifi2, the GUI only reflects 2 nodes (nifi1 and nifi2 but not nifi3).  https://nifi3:9337 only reflects 1 node i.e., nifi3.

I am using an external zookeeper and these are the properties I changed in nifi.properties:

• nifi.cluster.is.node = true
• nifi.cluster.load.balance.host = nifi[1,2,or 3]...depending on which host the nifi.properties file exists in
• nifi.zookeeper.connect.string = zookeeper1 private IP address:2181, zookeeper2 private IP address:2181, etc...

Additionally, per the directions, I updated the state-management.xml file and added the list of zookeeper instances in the Connect String property i.e., zoo1private IP address:2181, etc...

I also updated the authorizers.xml file, by adding the following:

in userGroupProvider section:

<property name="Initial User Identity 1">CN=ec2-user</property>

<property name="Initial User Identity 2">CN=nifi1, OU=NIFI</property>

<property name="Initial User Identity 3">CN=nifi2, OU=NIFI</property>
<property name="Initial User Identity 2">CN=nifi3, OU=NIFI</property>

in accessPolicyProvider section:

<property name="Initial Admin Identity">CN=ec2-user</property>

<property name="Node Identity 1">CN=nifi1, OU=NIFI</property>

<property name="Node Identity 2">CN=nifi2, OU=NIFI</property>
<property name="Node Identity 3">CN=nifi3, OU=NIFI</property>

I checked the logs of each of the nodes, and there weren't any WARNINGS or errors.

Any ideas/guidance on why all 3 nodes are not joining the cluster?

Thank you for any ideas/guidance

[#2]

@davehkd 

Parse the nifi-app.log for messages related to heartbeat and make sure that that all you nodes are creating and sending heartbeats to the ZK elected cluster coordinator.

Check the nifi-app.log on the node elected as the cluster coordinator (This would be either node 1 or 2 since they show 2/2 connected nodes) for heartbeat messages and you should see it receiving heartbeats from all three nodes.

If it is not receiving heartbeats from node 3, make sure their are no network or DNS resolution issue between node 3 and the other 2 nodes in the cluster.

Verify that their are no typos in the nifi.properties on node 3 in the following sections:
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#cluster_common_properties

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#cluster_node_properties

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#zookeeper-properties
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#web-properties

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security_properties

Check the nifi-user.log on the elected cluster coordinator and on node 3 for any TLS handshake exceptions.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#3]

Hello Matt, thanks for your input:

I checked the nifi-app.log on nifi1.

I did scan through looking for heartbeat messages in regard to nifi3, and only saw 2 identical separate entries:

Process Cluster Protocol Request - 25] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 7f0cacaa-... (type=NODE_STATUS_CHANGE, length=1069 bytes) from nifi3 in 174 millis

I had added all 3 nodes to the hosts file of each respective node, and tested that they could all successfully ping each other i.e., the hosts file entries are correct.

I took a look at the nifi-app.log for nifi2, and saw these entries:

Process Cluster Protocol Request-1]o.a.n.c.c.node.NodeClusterCoordinator No existing node with ID: 5bxxx; will add Node as NodeIdentifier[UUID=5bxxxx, API Address = nifi3:9443, Cluster Socket Address = nifi3:11443, Load Balance Address = nifi3:6342, Site-to-Site Raw Address = nifi3:10443, Site-to-Site HTTP Address = nifi.9443, Site-to-Site Secure = true, Node Identities =[]]

[Process Protocol Request-1] o.a.n.c.c.flow.PopularVoteFlowElection Vote cast by nifi3:9443; this flow now has 1 votes

...

I did not see any TLS issues in any of the logs.

I went through the nifi.properties and did not see any miskeys...

[#4]

@davehkd 

Unfortunately, I would need to have access to the nifi-app.log file(s) from each node to dig in deeper.
Did you copy the flow.xml.gz,  flow.json.gz, users.xml, and authorizations.xml files from NiFi node 1 or 2 to NIFi node 3?  These files all need to match in order for a node to join the cluster.  

1. The UI of nifi1 or nifi2  shows "2/2" in the status bar just along top of canvas?
2. The UI of nifi3 shows "1/1" in the status bar just along the top of the canvas?

If both above are true, this indicates nifi3 is member of a different cluster.
Possible result if issue with your ZK or using a different ZK root node (nifi.zookeeper.root.node).  Check for any leading or trailing whitespace in your configuration.

You may also want to inspect your ZK logs for the connections coming from all three nodes.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Matt