Support Questions
Find answers, ask questions, and share your expertise

NiFi - How to wait for processing of splitted FFs (WAIT and NOTIFY ?)

Expert Contributor

Hi, the problem to solve is demonstrated in the screenshot.

The response of the first InvokeHTTP provides 1 FF which is splitted to n FFs. Each FF gets processed and is send with the second InvokeHTTP. The further process has to wait until all single FFs are ready - with or without error.

I found this information https://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/ but I don't know whether it solves this problem and if so - how. Using NiFi 1.6.0

Thanks in advance for any help!

93192-hwc-wait-1.png

1 REPLY 1

Expert Contributor

OK, as far I have tested the MergeConent waits by default for ALL fragments of the original FF.

Because of the position of the MergeContent AFTER an InvokeHTTP there is always the possibility that at least one fragmented FF goes to Retry or Failure. So waiting for ALL fragments is NOT possible.

Lets assume I have 2 fragments of 1 original FF. One goes after InvokeHTTP to Response one goes to Failure.

If I use the property "Max Bin Age" at MergeContent e.g. 10 sec the fragment which came to Response is directed to Failure of the MergeConent with this note in bulletin board:

MergeContent[id=c7cb1d1f-325f-1427-3f72-e124b4d8902d] Cannot Defragment FlowFiles with Fragment Identifier 31604637-58ca-4b37-9eaf-bd66c4815a57 because the expected number of fragments is 2 but found only 1 fragments [...]

Is there a chance to handle this, merging only the fragments which come to Response within the specified time?

Or is there a complete other way to handle this?

Any hint is appreciated!