question Re: Unable to get Nifi site-to-site RPG to balance across nodes in Support Questions

Unable to get Nifi site-to-site RPG to balance across nodes

Jim_B — Sun, 18 Aug 2019 01:00:27 GMT

Trying to get nifi site-to-site RPG (loopback to a single cluster) to balance across nodes. Real flow sends out table names for processing across node, but put together a simple test flow to test. Still only going to one node after setting the usual suspects of properties

Attachment 1 shows Test flow. RPG url is set to node xxx22, one of the worker nodes

Attachment 2 shows settings for the test input port (Concurrent tasks 7, Batch Settings count 1)

Attachment 3 shows Concurrent tasks of receiving process (not sure this matters, but I was desparate!)

Attachment 3 shows provenance of receiving process with all files going to node xxx28.

Anyone see an obvious (or not so obvious!) configuration error here?

Thanks!

Re: Unable to get Nifi site-to-site RPG to balance across nodes

Jim_B — Fri, 24 Aug 2018 06:13:59 GMT

Did some additional research and the key to this is setting the following in the Advanced nifi-properties.

nifi.remote.input.host={{nifi_node_host}}

The documentation is a little weak, but setting this to a value, causes only that particular node to listen for RPG requests. Setting to this variable, will substitute each host's name in its' own nifi.properties file, and you are good to go. Still have to set the levels of concurrency for the overall canvas, and processors's, of course. But this gets all the nodes to listen for remote requests.

Re: Unable to get Nifi site-to-site RPG to balance across nodes

Jim_B — Fri, 24 Aug 2018 06:17:44 GMT

After some research and help, I found that I had incorrectly set the following nifi.remote.input.host property. It should be set as follows in Advance nifi properties:

nifi.remote.input.host={{nifi_node_host}}

Each node should have a different value for nifi.remote.input.host, it's the value the current node is going to advertise for s2s comms... if you set that the same on all nodes then they are all advertising the same hostname and thus all data going to same host.

You have to set the other multi-threading parameters such as "Maximum Timer Driven Thread Count" in controller settings, and "Concurrent Tasks" in the appropriate processors. But, this gets multiple nodes to listen for the RPG requests.