configuring the nifi.remote.input.host for the same cluster and nifi ui

[#1]

Hi All,

Thanks a lot this awesome community.

I am trying to balance the load after the list files using fetch files. In this case the listfile and fetchfile are on the same nifi ui.

I need to set this

nifi.remote.input.host=<fqdn of all the nodes in the cluster?>

Do I need to give the FQDN of all the nifi nodes in a comma separated way? any suggestions

Thanks a lot

Dheeru

[#2]

@dhieru singh

You should set that property to the hostname for each of the nodes in the cluster.

So, for node1, it would be nifi.remote.input.host=node1, and node2 would be nifi.remote.input.host=node2 and so on for each node.

Then you'll drop a Remote Process Group on the UI. Have you ever used a Remote Process Group before?

[#3]

@Wynner Thanks a lot for the response, appreciate it. so I am not sure if I understood completely, I will make this change through ambari UI right? and if I make change thru ambari UI, I need to go on each node and make the change. for example

on node1

nifi.remote.input.host=node1

on node 2

nifi.remote.input.host=node2

No I have never used the remote process group, I was looking to distribute the load on same the nifi ncm using remote process group, any suggestion, advise?

Thanks Dheeru

[#4]

@dhieru singh

The flow would look something like this:

The output from the ListSFTP would go to the Remote Process Group which has the NiFi URL of one of the nodes.

There will need to be an inport port at the root level of the NiFi UI, you will use to connect to from the ListSFTP

And then the output from that port will go to the FetchSFTP, which in my example is in a process group

Do you follow?

[#5]

Should the RPC URLs list multiple node? (i.e. http://nifi_node1:9091/nifi,http://nifi_node2:9091/nifi,http://nifi_node3:9091/nifi)

Only reason i ask is that I have a simular set up for list/fetch S3 processors and noticed that it was all being processed via a single node even though i have a 3 node cluster.

Thank you

JF

[#6]

@Jose Gonzalez

You can specify more then one host but it is not required. Once the RPG establish a connection to the target host it will retrieve the S2S details of the target cluster and store that locally. If the host you provided become unavailable at anytime after that initial connection , it will try anyone of the other nodes it learned about previously to get S2S details. Having multiple nodes configured helps when NiFi by giving the source Nifi more then one target node to establish initial connection with.

-

Your load-balancing issue is completely unrelated to how many nodes URLS you configured in your RPG. Here is an article that covers how load-balancing works with an RPG:
https://community.hortonworks.com/content/kbentry/109629/how-to-achieve-better-load-balancing-using-...

-

Thanks,

Matt