Skip to main content
gmgiadinh176
gmgiadinh176
Bronze ⭐️⭐️⭐️
September 25, 2022
Question

NIfi - activate flow file one by one

Hello I am trying to make a data flow through Nifi which are listFile -> fetchFile -> data transformation -> putKudu

 

However, I want to somehow pause the fetchFile processor or hold on to the flowfiles received from the local server until a previous flowfile completely goes through the downstream, after putKudu processor, then release the next flowfile one by one.

I looked at Wait and Notify processors but could not find a way to notify the next flowfile in the queue to be released.

 

Is it possible to apply this logic in the Nifi without using custom processor?

 

Any help is appreciated.

2 replies

SAMSAL
SAMSAL
Platinum ⭐️⭐️⭐️
September 25, 2022

Hi,

 

Have you looked into the ControlRate processor. Its not exactly doing what you want but it allows you to put a time to hold a flowfile before it gets released to the next processor. For example if you think that you need a second to process one flowfile from the moment its fetched to the moment its processed by PutKudu, then you can configure the ControlRate as follows:

SAMSAL_0-1664114278646.png

 

Translate
Faerballert
Faerballert
Gold ⭐️
September 26, 2022

Hello,

 

you can configure that just 1 Flowfile will be handled inside of a specific ProcessorGroup. 

There is the config option on PG which is called: Process Group FlowFile Concurrency

for that you can set the value: Single FlowFile per Node

 

After PutKudu you will destroy the FlowFile or route it out from ProcessorGroup, then the next FlowFile will be released to enter to ProcessorGroup

 

In your case the Flow would look like:

ListFile Processor -> ProcessorGroup (handels fetchFile, data Transformation and putKudu)

Translate
Terms & ConditionsAccessibility statement