Load distribution of nifi flow files adding new node to remote process group

[#1]

I have flow files running through a nifi cluster. A remote process group is used to basically load balance flow files. if I added a new node to the cluster (while flow files are running, no stoppage), will nifi automatically distribute the load to the new nifi node? Does the cluster coorindator perform this function?

[#2 · Accepted solution]

@Sunile Manjee

As part of the S2S protocol, the RPG will receive continuously get updated list of available peers (which includes load on peers in terms of total queued flowfiles) which it will store locally. It will then load-balanced based on that information. If for some reason it fails to get an update at any time it will continue to try and deliver and load-balance using the last list of known peers and loads.

The RPG can get an updated list of available peers from any one of the target NiFi cluster's nodes. It only must get that list from the configured URL on the RPG the first time it connects. After that is has a list of peers it can get that info from.

Thanks,

Matt

[#3]

@Matt Clarke I believe this answers my question but need to confirm. New nodes to the remote cluster can be added real time without any downtime and the RPG will be updated with list of all available nodes including any new nodes added. Again important this happens real time without any downtime. please confirm.

[#4]

Yes, nodes can be added in real time to an existing niFi Cluster.

Yes, the RPG is constantly get an updated list of nodes in the target cluster. That includes if a node was added or removed.

[#5]

Thanks for sharing this worthwhile information here. Just a great article or information. Hope you will sharing new article or post in future. Good blog post. I want to thank you for interesting Register LLp In India and helpful information and I like your point of view. Thank you! I love to read this type of material Good and attractive information I take from it. Thank you for posting such a good