Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

nifi RPG

avatar
New Contributor

Hello all!
Is there a way to use PGR and input/output ports in the same nifi instance?
The aim of that is to reuse the same processor multiple times by only reference to one PGR.
If not, are there any other suggestions to achieve the same idea؟

1 ACCEPTED SOLUTION

avatar
Master Mentor

@I_Mesh01 

You certainly can use Remote Process Group (RPG) to send FlowFiles to a Remote Input Port on the same NiFi deployment (standalone of cluster), but would not be something I would recommend.  The RPG is designed to send FlowFiles over the network to another NiFi instance.  Even when used to send FlowFile to same NiFi, it will be sending FlowFiles over the network.

I think your design intent is to have many RPGS all having some remote Input Port that feeds some shared processor.  Then what do you do with the FlowFiles tat come out of the pone processor (singular downstream flow?)

If that is the case you would be better off  creating yoru many dtaflows in separate easy to manage Process Groups (PGs) and use non Remote Output ports in all of those PGS to feed to a Non Remote input port in a single PG with your shared downstream dataflow.

I would recommend against the many Remote Output ports being pulled from by a single RPG design approach completely as it is not efficient in NiFi cluster as you can end up with the RPG on one node pulling bulk of the remote output port queued FlowFiles (not very distributed). RPGs handle the distribution of FlowFiles to Cluster nodes in a much more balance way when the ROG is pushing/sending FlowFiles to a Remote Input ports.  But as i mentioned above i would not add the network layer if possible in your design.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt



View solution in original post

3 REPLIES 3

avatar
Community Manager

@I_Mesh01 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts   @steven-matison and @SAMSAL who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
Master Mentor

@I_Mesh01 

You certainly can use Remote Process Group (RPG) to send FlowFiles to a Remote Input Port on the same NiFi deployment (standalone of cluster), but would not be something I would recommend.  The RPG is designed to send FlowFiles over the network to another NiFi instance.  Even when used to send FlowFile to same NiFi, it will be sending FlowFiles over the network.

I think your design intent is to have many RPGS all having some remote Input Port that feeds some shared processor.  Then what do you do with the FlowFiles tat come out of the pone processor (singular downstream flow?)

If that is the case you would be better off  creating yoru many dtaflows in separate easy to manage Process Groups (PGs) and use non Remote Output ports in all of those PGS to feed to a Non Remote input port in a single PG with your shared downstream dataflow.

I would recommend against the many Remote Output ports being pulled from by a single RPG design approach completely as it is not efficient in NiFi cluster as you can end up with the RPG on one node pulling bulk of the remote output port queued FlowFiles (not very distributed). RPGs handle the distribution of FlowFiles to Cluster nodes in a much more balance way when the ROG is pushing/sending FlowFiles to a Remote Input ports.  But as i mentioned above i would not add the network layer if possible in your design.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt



avatar
Community Manager

@I_Mesh01 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  Thanks.


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community: