question Re: nifi RPG in Support Questions

nifi RPG

I_Mesh01 — Fri, 14 Jul 2023 14:48:34 GMT

Hello all!
Is there a way to use PGR and input/output ports in the same nifi instance?
The aim of that is to reuse the same processor multiple times by only reference to one PGR.
If not, are there any other suggestions to achieve the same idea؟

Re: nifi RPG

DianaTorres — Fri, 14 Jul 2023 20:52:56 GMT

@I_Mesh01 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @steven-matison and @SAMSAL who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: nifi RPG

MattWho — Fri, 14 Jul 2023 21:19:26 GMT

@I_Mesh01

You certainly can use Remote Process Group (RPG) to send FlowFiles to a Remote Input Port on the same NiFi deployment (standalone of cluster), but would not be something I would recommend. The RPG is designed to send FlowFiles over the network to another NiFi instance. Even when used to send FlowFile to same NiFi, it will be sending FlowFiles over the network.

I think your design intent is to have many RPGS all having some remote Input Port that feeds some shared processor. Then what do you do with the FlowFiles tat come out of the pone processor (singular downstream flow?)

If that is the case you would be better off creating yoru many dtaflows in separate easy to manage Process Groups (PGs) and use non Remote Output ports in all of those PGS to feed to a Non Remote input port in a single PG with your shared downstream dataflow.

I would recommend against the many Remote Output ports being pulled from by a single RPG design approach completely as it is not efficient in NiFi cluster as you can end up with the RPG on one node pulling bulk of the remote output port queued FlowFiles (not very distributed). RPGs handle the distribution of FlowFiles to Cluster nodes in a much more balance way when the ROG is pushing/sending FlowFiles to a Remote Input ports. But as i mentioned above i would not add the network layer if possible in your design.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Re: nifi RPG

DianaTorres — Tue, 18 Jul 2023 00:14:18 GMT

@I_Mesh01 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.