NIFI: Node discovering for Cluster

[#1]

Hi,

For example, I have 3 NiFi nodes connected to one cluster.

If I'll have additional 2 NiFi nodes, is there a way to "ask them to join" cluster?

All nodes in one private network.

Thank you for your help.

[#2 · Accepted solution]

@Gu Gur

If you configure the two nodes the same way as the current three nodes in the cluster, they should "ask to join" automatically.

Which version of NiFi are you using?

[#3]

1.0.1

but how will they find master node?

it means that I need to reconfigure all 5 nifi nodes in order to work?

[#4]

@Gu Gur

No, you only have to configure the two new nodes to be part of the cluster that is already running. Make sure to configure to use the same root node in zookeeper, set this property nifi.zookeeper.root.node, to the same values that the other three nodes are using, and of course configure the cluster node properties.

[#5]

Hi, Thank you very much for answer. So can I implement it using ExecuteScript processor ? Could you please give some tips where to look at.

[#6]

@Gu Gur

You could you the ExecuteScript processor, but that would probably be more work than just doing it manually since it is only two nodes you are adding.

I am assuming you have not secured the cluster.

You could just copy the conf directory over to the new nodes, and then update the all entries to match the name of the new host.

For example, assuming the current nodes are node1.localdomain, node2.localdomain and node3.localdomain, then copy node3's conf directory to a new node you want to add, you could just change everywhere you have node3 to node4, or whatever is the new node system name.

Are you following me?

[#7]

@Gu Gur

I agree, that would be a nice tool. But a difficult task because there are unique environments for almost each user and multiple environments for each user.

[#8]

@Gu Gur

You just need to make sure the following lines in the new nodes nifi.properties file match the same lines in the nifi.properties files of those nodes already in your cluster.

I also recommend setting/updating the following line in the nifi.properties file on every node in your cluster each time you add or remove a node:

nifi.cluster.flow.election.max.candidates=

It is blank by default which means the UI will be unavailable for 5 minutes whenever the cluster is restarted while NiFi waits for the leader election to complete.

Of course all your rest of the "cluster node properties" must be set on your new nodes as well.

Any changes to the nifi.properties file will require a NiFi restart to take affect

Thanks,

Matt

[#9]

Yes, I follow you ) But it's just example 2 more new nodes. It could be 10-15 and so on...

So in this case it would be cool to automate this process somehow.