- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
nifi RPG
- Labels:
-
Apache NiFi
Created 07-14-2023 07:48 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello all!
Is there a way to use PGR and input/output ports in the same nifi instance?
The aim of that is to reuse the same processor multiple times by only reference to one PGR.
If not, are there any other suggestions to achieve the same idea؟
Created 07-14-2023 02:19 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@I_Mesh01
You certainly can use Remote Process Group (RPG) to send FlowFiles to a Remote Input Port on the same NiFi deployment (standalone of cluster), but would not be something I would recommend. The RPG is designed to send FlowFiles over the network to another NiFi instance. Even when used to send FlowFile to same NiFi, it will be sending FlowFiles over the network.
I think your design intent is to have many RPGS all having some remote Input Port that feeds some shared processor. Then what do you do with the FlowFiles tat come out of the pone processor (singular downstream flow?)
If that is the case you would be better off creating yoru many dtaflows in separate easy to manage Process Groups (PGs) and use non Remote Output ports in all of those PGS to feed to a Non Remote input port in a single PG with your shared downstream dataflow.
I would recommend against the many Remote Output ports being pulled from by a single RPG design approach completely as it is not efficient in NiFi cluster as you can end up with the RPG on one node pulling bulk of the remote output port queued FlowFiles (not very distributed). RPGs handle the distribution of FlowFiles to Cluster nodes in a much more balance way when the ROG is pushing/sending FlowFiles to a Remote Input ports. But as i mentioned above i would not add the network layer if possible in your design.
If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.
Thank you,
Matt
Created 07-14-2023 01:52 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@I_Mesh01 Welcome to the Cloudera Community!
To help you get the best possible solution, I have tagged our NiFi experts @steven-matison and @SAMSAL who may be able to assist you further.
Please keep us updated on your post, and we hope you find a satisfactory solution to your query.
Regards,
Diana Torres,Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
Created 07-14-2023 02:19 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@I_Mesh01
You certainly can use Remote Process Group (RPG) to send FlowFiles to a Remote Input Port on the same NiFi deployment (standalone of cluster), but would not be something I would recommend. The RPG is designed to send FlowFiles over the network to another NiFi instance. Even when used to send FlowFile to same NiFi, it will be sending FlowFiles over the network.
I think your design intent is to have many RPGS all having some remote Input Port that feeds some shared processor. Then what do you do with the FlowFiles tat come out of the pone processor (singular downstream flow?)
If that is the case you would be better off creating yoru many dtaflows in separate easy to manage Process Groups (PGs) and use non Remote Output ports in all of those PGS to feed to a Non Remote input port in a single PG with your shared downstream dataflow.
I would recommend against the many Remote Output ports being pulled from by a single RPG design approach completely as it is not efficient in NiFi cluster as you can end up with the RPG on one node pulling bulk of the remote output port queued FlowFiles (not very distributed). RPGs handle the distribution of FlowFiles to Cluster nodes in a much more balance way when the ROG is pushing/sending FlowFiles to a Remote Input ports. But as i mentioned above i would not add the network layer if possible in your design.
If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.
Thank you,
Matt
Created 07-17-2023 05:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@I_Mesh01 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
Regards,
Diana Torres,Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
