@Zifo1
When using Site-to-SIte via Remote Process Groups (RPG) and Remote Input or Output ports between NiFi clusters, it is most efficient to push rather then pull data (FlowFiles).
The NiFi RPG always acts as the client side of the connection. It will either send FlowFiles to a Remote Input Port or fetch FlowFiles from a Remote Output port. I would avoid fetching from Remote Output ports. You get better FlowFiles distribution across teh destination cluster when you send FlowFiles from the RPG.
If the FlowFiles traverse both directions, you would simply setup a RPG on both NiFi clusters to push FlowFiles to the Remote Input Ports on opposite clusters.
Details about Site-To-Site can be found here:
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site
As far as the RPG goes, I recommend using the "RAW" transport protocol over HTTP. RAW requires that the dedicated RAW port is configured in the server side NiFi's nifi.properties file. RAW establishes a raw socket connection on the dedicated configured port. HTTP utilizes the same HTTPS port that all other NiFi interactions use. You'll need to make sure the network connectivity exists between both your NiFi Clusters on both the HTTP(s) and RAW ports. HTTP is always used to fetch Site-to-Site Details.
Setting up the client side (Remote Process Group) Documentation is here:
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#configure-site-to-site-client-nifi-insta...
Setting up the sever side (NiFi with Remote Input or Remote Output ports) documentation can be found here:
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#configure-site-to-site-server-nifi-insta...
Even with Site-To-Site the communications between the two NiFi clusters requires both authentication and authorization. Authentication is established via a mutual TLS handshake initiated by the RPG. For Site-to-Site, the keystore and truststore setup en each NiFi's nifi.properties file are used in the MutualTLS exchange.
NOTE: The NiFi out-of-box auto generated keystores and truststores are not suitable for negotiating a successful Mutual TLS handshake.
There are numerous authorization policies that must be setup on the server side (remote ports NiFi) so that the client side (NiFi with RPG) is able to successfully send FlowFiles over Site-to-Site:
1. Retrieve Site-to-Site Details - This policy authorizes the client NiFi nodes (so all nodes in the client side NiFi cluster must be authorized) to retrieve site-to-site details from the server side NiFi. This includes details like number of nodes, load on those nodes, authorized remote ports, site-to-site raw port, https port, etc.
2. Receive data via Site-To-Site - This policy is setup on Remote Input ports to authorize the client side NiFi nodes to send FlowFiles to this specific port.
3. Send data via Site-to-Site - This policy is setup on the Remote Output Ports and allows authorized client nodes to fetch FlowFiles from the Remote output port.
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
