Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How send data from nifi to minifi ( same config)

avatar
New Contributor

Dears,

How send data from nifi cluster to 3 minifi ( same config )

 

"Data" -> NiFi Cluster -> RPG ->  MiNiFi 1, MiNiFi 2, MiNiFi 3 (receive same "Data" in 3 minifi)

 

Thank you in advance.

Best Regards,

1 ACCEPTED SOLUTION

avatar
Super Mentor

@Submar 

 

MiNiFi is not capable of receiving data over S2S being sent to it by another NiFi instance running a Remote Process Group (RPG).  It is not something that is supported nor recommended even if it was.

S2S was built as a way to smartly distribute FlowFiles being sent from one NiFi instance to a cluster of NiFi instances.  Since MiNiFi is not a cluster, the overhead associated with fetching S2S details makes no sense.

Now you could set up a RPG on your MiNiFi that pulls from a remote outputPort on your NiFi, but again, that is not an ideal or recommended solution. It also would not result in same data landing on every MiNiFi since data is consumed by which ever MiNiFi connects first to the NiFi output port.

If the goal here is to send "FlowFiles" (FlowFile metadata/attributes and FlowFile content) from NiFi to MiNiFi, the better option would be to use the ListenHTTP processor on each of your MiNiFi instance and then have a separate PostHTTP processor for each of your 3 MiNiFi instances.  Then have the same FlowFiles routed to each of those three postHTTP processors.   The PostHTTP processor also has a property "Send as FlowFile" which when set to true will send the entire FlowFile and not just the content to the target MiNiFi.  The flows on all 3 of your MiNiFi instances would still be identical with each listening on the same configured listenHTTP port. 

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

View solution in original post

2 REPLIES 2

avatar
Super Mentor

@Submar 

 

MiNiFi is not capable of receiving data over S2S being sent to it by another NiFi instance running a Remote Process Group (RPG).  It is not something that is supported nor recommended even if it was.

S2S was built as a way to smartly distribute FlowFiles being sent from one NiFi instance to a cluster of NiFi instances.  Since MiNiFi is not a cluster, the overhead associated with fetching S2S details makes no sense.

Now you could set up a RPG on your MiNiFi that pulls from a remote outputPort on your NiFi, but again, that is not an ideal or recommended solution. It also would not result in same data landing on every MiNiFi since data is consumed by which ever MiNiFi connects first to the NiFi output port.

If the goal here is to send "FlowFiles" (FlowFile metadata/attributes and FlowFile content) from NiFi to MiNiFi, the better option would be to use the ListenHTTP processor on each of your MiNiFi instance and then have a separate PostHTTP processor for each of your 3 MiNiFi instances.  Then have the same FlowFiles routed to each of those three postHTTP processors.   The PostHTTP processor also has a property "Send as FlowFile" which when set to true will send the entire FlowFile and not just the content to the target MiNiFi.  The flows on all 3 of your MiNiFi instances would still be identical with each listening on the same configured listenHTTP port. 

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

avatar
New Contributor

I understand and got the solution to this problem then thank you for your help.

Thanks MattWho.