Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to get Nifi site-to-site RPG to balance across nodes

Solved Go to solution
Highlighted

Unable to get Nifi site-to-site RPG to balance across nodes

Expert Contributor

Trying to get nifi site-to-site RPG (loopback to a single cluster) to balance across nodes. Real flow sends out table names for processing across node, but put together a simple test flow to test. Still only going to one node after setting the usual suspects of properties

86600-picture1.png

Attachment 1 shows Test flow. RPG url is set to node xxx22, one of the worker nodes

Attachment 2 shows settings for the test input port (Concurrent tasks 7, Batch Settings count 1)

Attachment 3 shows Concurrent tasks of receiving process (not sure this matters, but I was desparate!)

Attachment 3 shows provenance of receiving process with all files going to node xxx28.

Anyone see an obvious (or not so obvious!) configuration error here?

Thanks!

86603-picture4.png

86602-picture3.png

86601-picture2.png

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Unable to get Nifi site-to-site RPG to balance across nodes

Expert Contributor

After some research and help, I found that I had incorrectly set the following nifi.remote.input.host property. It should be set as follows in Advance nifi properties:

nifi.remote.input.host={{nifi_node_host}}

Each node should have a different value for nifi.remote.input.host, it's the value the current node is going to advertise for s2s comms... if you set that the same on all nodes then they are all advertising the same hostname and thus all data going to same host.

You have to set the other multi-threading parameters such as "Maximum Timer Driven Thread Count" in controller settings, and "Concurrent Tasks" in the appropriate processors. But, this gets multiple nodes to listen for the RPG requests.

View solution in original post

2 REPLIES 2
Highlighted

Re: Unable to get Nifi site-to-site RPG to balance across nodes

Expert Contributor

Did some additional research and the key to this is setting the following in the Advanced nifi-properties.

nifi.remote.input.host={{nifi_node_host}}

The documentation is a little weak, but setting this to a value, causes only that particular node to listen for RPG requests. Setting to this variable, will substitute each host's name in its' own nifi.properties file, and you are good to go. Still have to set the levels of concurrency for the overall canvas, and processors's, of course. But this gets all the nodes to listen for remote requests.

Re: Unable to get Nifi site-to-site RPG to balance across nodes

Expert Contributor

After some research and help, I found that I had incorrectly set the following nifi.remote.input.host property. It should be set as follows in Advance nifi properties:

nifi.remote.input.host={{nifi_node_host}}

Each node should have a different value for nifi.remote.input.host, it's the value the current node is going to advertise for s2s comms... if you set that the same on all nodes then they are all advertising the same hostname and thus all data going to same host.

You have to set the other multi-threading parameters such as "Maximum Timer Driven Thread Count" in controller settings, and "Concurrent Tasks" in the appropriate processors. But, this gets multiple nodes to listen for the RPG requests.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here