Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What is the best way to receive a NIFI flowfile and asynchronously generate another driven by the input flowfile

What is the best way to receive a NIFI flowfile and asynchronously generate another driven by the input flowfile

New Contributor

I am looking to create a customNIFI processor. It receives flowfiles, but in order to generate the resulting flowfile, it must first reach out to a backend service to get information that must be added to the output flowfile.


I am currently thinking of accessing the backend service in a worker thread in the custom processor. Once the additional information is received, I'd then queue this for an asynchronous creation of an output flowfile (in onTrigger).


Is this a reasonable thing to do in a custom processor? Typically (using the egress pattern - or many of the other common patterns- documented on the NIFI developer page), input flowfiles are immediately (synchronously) transformed into an output flowfile.


However that is not the pattern I have in mind.. Input flowfiles will instead be asynchrnously transformed into output flowfiles (and this might not even be a 1 to 1 mapping from input flowfile to output flowfile). In this regard the processor will have behaviour similar to MergeContent, or MergeRecord (which are indeed asynchronously generating output flowfiles).