For NiFi S2S, is it better to us load balancer hostname or separate config groups with individual NiFi hostnames in nifi.remote.input.host

[#1]

I am using Ambari to set up NiFi S2S to use with RPGs, and I have a question about the config 'nifi.remote.input.host'. I have a loadbalancer (HAProxy) in front of my 2 NiFi UI's to load balance the access to the UI's (for port 9091 since SSL is enabled). Am I better to use that host in the 'nifi.remote.input.host' config, or should I be creating separate NiFi config groups per NiFi node and putting in each NiFi host into that config?

[#2 · Accepted solution]

Hi @Chad Woodhead,

This property needs to be set if and only if you are doing S2S using RAW protocol. Any reason not to use HTTP protocol for S2S? This way, you don't have anything to set and it'll work OOTB with default configuration.

If, however, you want to use RAW and you are configuring NiFi using Ambari, you don't need to use config groups, you can set the property to:

{{nifi_node_host}}

It's a placeholder that contains the FQDN of each node.

Note that this property already has a default value generated if you let it blank (see the description of this property in Ambari). Unless you have a specific network configuration, the only property you would need to specify is nifi.remote.input.socket.port to define the port that will be used for RAW S2S.

Hope this helps.

[#3]

Hi @Pierre Villard,

Thanks for the help! '{{nifi_node_host}}' worked for getting RAW to work.

As far as choosing HTTP or RAW, I was reading some posts by @Matt Clarke and was choosing RAW based on the following comments by him:

When using the RAW format (Socket based transfer), the "nifi.remote.input.host" and "nifi.remote.input.socket.port" configured values from each of the target NiFi instances are used by the NiFi client as the destination for sending FlowFiles.

When using the HTTP format, the "nifi.remote.input.host" and the "nifi.web.http.port" or "nifi.web.https.port" configured values from each of the target NiFi instances are used by the NiFi client as the destination for sending FlowFiles.

Advantage of RAW format is that their is a dedicated port for all S2S transfers, so under high load it affect on the NiFi HTTP interface is minimal.

Advantage of HTTP, you do not need to open an additional S2S port since teh same HTTP/HTTPS port is used to transfer FlowFile.

Having a separate port from the UI seemed like a good idea to me, especially in a production environment. Could you add on to what Matt has stated to any other advantages/disadvantages of using either RAW or HTTP?

[#4]

That's completely valid. Let's say that HTTP has the advantage of not requiring any new configuration when installing NiFi. In some environments, adding the use of an additional port can create additional work with network/security teams. Another advantage of HTTP is the possibility to use a proxy if that's required to allow communication between multiple sites.

If you expect to have high load with S2S and can manage the extra conf, RAW is certainly a better option.

Hope this helps.