question NIfi - activate flow file one by one in Support Questions

NIfi - activate flow file one by one

gmgiadinh176 — Sun, 25 Sep 2022 09:44:58 GMT

Hello I am trying to make a data flow through Nifi which are listFile -> fetchFile -> data transformation -> putKudu

However, I want to somehow pause the fetchFile processor or hold on to the flowfiles received from the local server until a previous flowfile completely goes through the downstream, after putKudu processor, then release the next flowfile one by one.

I looked at Wait and Notify processors but could not find a way to notify the next flowfile in the queue to be released.

Is it possible to apply this logic in the Nifi without using custom processor?

Any help is appreciated.

Re: NIfi - activate flow file one by one

SAMSAL — Sun, 25 Sep 2022 13:58:42 GMT

Hi,

Have you looked into the ControlRate processor. Its not exactly doing what you want but it allows you to put a time to hold a flowfile before it gets released to the next processor. For example if you think that you need a second to process one flowfile from the moment its fetched to the moment its processed by PutKudu, then you can configure the ControlRate as follows:

Re: NIfi - activate flow file one by one

Faerballert — Mon, 26 Sep 2022 14:08:31 GMT

Hello,

you can configure that just 1 Flowfile will be handled inside of a specific ProcessorGroup.

There is the config option on PG which is called: Process Group FlowFile Concurrency

for that you can set the value: Single FlowFile per Node

After PutKudu you will destroy the FlowFile or route it out from ProcessorGroup, then the next FlowFile will be released to enter to ProcessorGroup

In your case the Flow would look like:

ListFile Processor -> ProcessorGroup (handels fetchFile, data Transformation and putKudu)