- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
NIFI Site to Site connection between Clusters
- Labels:
-
Apache NiFi
-
Apache Zookeeper
Created 12-13-2024 04:03 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have 2 NIFI cluster, one NIFI cluster which is running as helm in EKS and another NIFI cluster which is deployed with 2 linux servers,
We want to enable Site to Site connection between EKS NIFI cluster to Remote NIFI cluster
what is the best way to connect via Remote Processor group, what configuration to be set for RPG
Created 12-13-2024 07:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Zifo1 Welcome to the Cloudera Community!
To help you get the best possible solution, I have tagged our NiFi experts @SAMSAL @mburgess who may be able to assist you further.
Please keep us updated on your post, and we hope you find a satisfactory solution to your query.
Regards,
Diana Torres,Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
Created 12-13-2024 07:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Zifo1 Welcome to the Cloudera Community!
To help you get the best possible solution, I have tagged our NiFi experts @SAMSAL @mburgess who may be able to assist you further.
Please keep us updated on your post, and we hope you find a satisfactory solution to your query.
Regards,
Diana Torres,Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
Created 12-13-2024 08:41 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Zifo1
When using Site-to-SIte via Remote Process Groups (RPG) and Remote Input or Output ports between NiFi clusters, it is most efficient to push rather then pull data (FlowFiles).
The NiFi RPG always acts as the client side of the connection. It will either send FlowFiles to a Remote Input Port or fetch FlowFiles from a Remote Output port. I would avoid fetching from Remote Output ports. You get better FlowFiles distribution across teh destination cluster when you send FlowFiles from the RPG.
If the FlowFiles traverse both directions, you would simply setup a RPG on both NiFi clusters to push FlowFiles to the Remote Input Ports on opposite clusters.
Details about Site-To-Site can be found here:
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site
As far as the RPG goes, I recommend using the "RAW" transport protocol over HTTP. RAW requires that the dedicated RAW port is configured in the server side NiFi's nifi.properties file. RAW establishes a raw socket connection on the dedicated configured port. HTTP utilizes the same HTTPS port that all other NiFi interactions use. You'll need to make sure the network connectivity exists between both your NiFi Clusters on both the HTTP(s) and RAW ports. HTTP is always used to fetch Site-to-Site Details.
Setting up the client side (Remote Process Group) Documentation is here:
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#configure-site-to-site-client-nifi-insta...
Setting up the sever side (NiFi with Remote Input or Remote Output ports) documentation can be found here:
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#configure-site-to-site-server-nifi-insta...
Even with Site-To-Site the communications between the two NiFi clusters requires both authentication and authorization. Authentication is established via a mutual TLS handshake initiated by the RPG. For Site-to-Site, the keystore and truststore setup en each NiFi's nifi.properties file are used in the MutualTLS exchange.
NOTE: The NiFi out-of-box auto generated keystores and truststores are not suitable for negotiating a successful Mutual TLS handshake.
There are numerous authorization policies that must be setup on the server side (remote ports NiFi) so that the client side (NiFi with RPG) is able to successfully send FlowFiles over Site-to-Site:
1. Retrieve Site-to-Site Details - This policy authorizes the client NiFi nodes (so all nodes in the client side NiFi cluster must be authorized) to retrieve site-to-site details from the server side NiFi. This includes details like number of nodes, load on those nodes, authorized remote ports, site-to-site raw port, https port, etc.
2. Receive data via Site-To-Site - This policy is setup on Remote Input ports to authorize the client side NiFi nodes to send FlowFiles to this specific port.
3. Send data via Site-to-Site - This policy is setup on the Remote Output Ports and allows authorized client nodes to fetch FlowFiles from the Remote output port.
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt