Sending data from NiFi to a specific MiNiFi Java

[#1]

Hi,

I want to send specific flowfiles to a specific remote MiNiFi Java.

Testing with 1 MiNiFi works great, out from NiFi, into MiNiFi for processing.

I'm prepping the MiNiFi flow on NiFi, templating, exporting, converting to yaml, modifying any security settings i need and restarting MiNiFi to accept the updated config.yml.

Now i'm adding additional MiNiFis.

On NiFi I've setup a PG with a RouteOnAttribute (Attribute is the specific MiNiFi name) to send to a specific S2S Output Port for that MiNiFi. Up to here, all seems fine.

The problem i have is in the MiNiFi template.

I have the RPG going into a processor, and that's when i specify the From Output Port which is specific to the MiNiFi.

How can i template/generalize/variabilize/other the From Output so that i don't need to make separate template for each and every MiNiFi instance?

I could theoretically move the RouteOnAttribute from the NiFi to the MiNiFi side but at that point it means i already sent a ton of data to all the MiNiFi instances instead of only to the one that's supposed to get it.

I figured i can update the config.yml with the specific connections IDs, but that seems like a broken process that's very prone to mistakes and maintenance for every change i make to the flow.

Does anyone have any pointers or tips for this situation?

[#2]

@benjbenj 

I know it has been a while. Have you found a solution for your use case?

[#3]

@Wynner i worked around this by having a process on the MiNiFi to update the local config.yml before starting. Basically i change out the connection source id for the processor i need. It's a fairly hacky way to make this work, but it's working and i haven't had the time or reason yet to find a cleaner way to get this done. if you find something, please come back here and comment. 🙂

[#4]

@benjbenj 

Would using a different type of transport be an option in your case?

Instead of using S2S you could possibly use a pair InvokeHTTP and ListenHTTP to send the data in a port that you can specify. The port wouldn't necessarily have to change between minifi instances, but you could also parameterize it if needed.

Cheers,

André

[#5]

@araujo that's an interesting way to go about it.

I haven't tried MTLS between InvokeHTTP and ListenHTTP, but i assume that's doable within the SSL context service.

At the moment, my MiNiFi clients connect periodically (every 10s) to see if there's anything queued for them from the NiFi side (each have their own output port in this config), i guess i would have to trigger that periodic fetch somehow...

You've given me stuff to think about. thanks. 🙂

[#6]

@benjbenj 
The ListenHTTP processor sets up a Listener on the configured port in the processor.  So there is not fetch involved in this flow design.  On your source NiFi, you would have a separate InvokeHttp processor for each unique MiNiFi agent you have deployed.  So while your data ingestion is setup on the same port on each unique MiNiFi, your invokeHTTP processors are each configured with a different target hostname.  When the invokeHTTP receive a FlowFile, it will connect to the http listener and transmit that FlowFile.

Thanks,

Matt

[#7]

@MattWho
Thanks Matt,
I understand what you're saying. in my case it's the other way around, I have clients (MiNiFi) initiating the connections when they want to send or receive data. This helps me deal with remote firewalls not allowing incoming connections and sporadic availability as they're the ones starting the connection.

At the moment, with S2S, The MiNiFi's connect periodically (every 10s) to see I there are any flowfiles waiting for them. I would need to figure out how to periodically generate that action to perform the same flow.

Example scenarios:
1. Client (MiNiFi side) is running a tail on a file. A new line comes into the file and needs to be sent over to NiFi for processing.
2. NiFi wants to send a file to the remote MiNiFi running a putfile processor. 
I have both scenarios working right now with S2S with no issues to a bunch of MiNiFi's, it's just that this requires hacky workarounds (scripts to modify the MiNiFi yml) to make this work and I would love to figure out a cleaner solution.