I am working with Nifi 1.0.0 and am working with 5 nodes, and want them to form a cluster. When I start them up, two or three of them form a cluster with the others forming their own individual clusters. Since Nifi 1.0.0 does not need a cluster manager to form a cluster, I did not elect one. Should I? I have seen it done in older versions but not for 1.0.0. Can you walk me through it?
I have updated the state-management.xml to reflect my zookeeper instances, I have updated nifi.properties for site-to-site properties, cluster properties, etc. I have updated the zookeeper.properties and authorizers.xml file to reflect the hostnames of all five nodes.
Can you help?
Under Cluster Node Properties, set the following:
nifi.cluster.flow.election.max.candidatesproperty, the cluster will not wait this long. The default is 5 minutes. Note that the time starts as soon as the first vote is cast.
Make sure they are all in the same zookeeper, same network and can talk on all ports to each other
I have set these values that you mentioned, and even updated the zookeeper.properties, state-management.xml , and authorizers.xml files with the appropriate node/port information, but the clustering now goes two sets of two nodes and then one node that does not form a cluster.
The first thing is to configure the list of the ZK (ZooKeeper) instances in the configuration file ‘./conf/zookeep.properties‘. Since our three NiFi instances will run the embedded ZK instance, I just have to complete the file with the following properties:
server.1=node-1:2888:3888 server.2=node-2:2888:3888 server.3=node-3:2888:3888
Then, everything happens in the ‘./conf/nifi.properties‘. First, I specify that NiFi must run an embedded ZK instance, with the following property:
I also specify the ZK connect string:
As you can notice, the ./conf/zookeeper.properties file has a property named dataDir. By default, this value is set to ./state/zookeeper. If more than one NiFi node is running an embedded ZK, it is important to tell the server which one it is.
I did all of these and configured the state-management.xml, zookeeper.properties, nifi.properties, and authorizers.xml files.
After much experimentation, I got all five nodes to cluster, and here is what I did from a high level.
1. Get two nodes to cluster
2. For every node thereafter, start and restart each node until the node "joins" the cluster
It can be time consuming but it worked.
Additionally, because you are using an embedded ZooKeeper, I would ensure that the conf/zookeeper.properties has the same values on all nodes for the server.1, server.2, ... server.N properties as @Timothy Spann mentioned above, and that all nodes that have the nifi.state.management.embedded.zookeeper.start property of nifi.properties are also mentioned as server.xx (i.e., if all 5 NiFi nodes have nifi.state.management.embedded.zookeeper.start set to true, then you should have server.1, server.2, server.3, server.4, server.5 in your zookeeper.properties file and in your nifi.properties connect string. It's also important to ensure that each node is able to reach all other nodes, as ZooKeeper can become pretty unhappy when one node is unable to communicate with other nodes.
Does this help?
@mpayne, I have five zookeeper instances running and they are configured properly in the zookeeper.properties file. I am not using the embedded zookeepers since I am forming a cluster, and using the embedded zookeepers for each node is not allowed. I never designated a root node since the documentation did not require it. Should I be doing this?
I will look into ensuring that all nodes can communicate with all zookeepers.
Thank you for the advice!