NiFi Site-to-Site example

Hi @learner-loading. I found this older community article that may be helpful. Site-To-Site communication between secured (HTTPS) and unsecured (HTTP) NiFi clusters

@learner-loading  I did some internal digging on this topic and came up with this too:

https://docs.cloudera.com/cfm/2.1.3/site-to-site/cdf-datahub-site-to-site.pdf

This is more modern doc on using CFM to accomplish site to site with NIFI.

@learner-loading 

The Site-To-Site functionality in NiFi has not evolved a whole lot since it was first introduced. 

You have an initial client server connection over http(s) followed by FlowFile transfer via either the same HTTP(s) connection or via a dedicated Raw socket connection depending on the configuration on the server side of the connection.

To use a mutual TLS HTTPS connection for initial connection, the following property must be set tp true:

nifi.remote.input.secure=true

In order to set this to true, NiFi must be configured with a keystore and truststore in the nifi.properties file as well.

To support HTTP(s) FlowFile transfer the following property must be set to "true":

nifi.remote.input.http.enabled=true

To support RAW socket connection for FlowFile transfer, the following property must be configured with   a port not in use by all nodes in yoru NiFi cluster:

nifi.remote.input.socket.port=10443

The Remote Process Group is always the "client" in the site-to-site connection.
The Remote Input and Remote output ports are then the "server" side of the connection.

So when the Remote Process Group (RPG) has been configured with a URL(s) (If target NiFi is a cluster setup, you can optionally provide a comma separated list of URLs for the hosts  in the same target cluster to offer some redundancy in case one of the target nodes is down), the RPG will start automatically connecting to first URL to try to fetch Site-To-Site (S2S) details about the server side of this connection.  Those server details include if the server side is configured secure or unsecured, if it supports HTTP, RAW, or both, the URLs for all connected nodes in target cluster, NiFi HTTTP port for all connected nodes, RAW port for all connected nodes, total amount of queued FlowFiles on each connected node, Available Remote Input and Output ports on target NiFi cluster.

With an HTTPS enabled Site-To-Site, the keystore and truststore files configured on both source/client Nifi and target/server NiFi will be used in the mutual TLS handshake connection made (The client certificate DN is used as the client identity passed to the server that needs to be properly authorized on the target Remote Input and Output ports.  If authorization is successful and the ROPG has "enabled transmission" FlowFiles will be able to transfer from RPG to Remote Input Port or  From a Remote Output Port to the RPG.  Better load-balancing is achieved with RPG pushing to Remote Input Ports.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

@learner-loading were you able to resolve your issue? If any of the above posts were the solution please mark the appropriate, as it will make it easier for others to find the answer in the future. 

mark as solution button