How to restrict NIFI processor to process only one flowfile from incoming queue at a time and complete its task and then continue with another flowfile in the incoming queue?

[#1]

Hello,

In my scenario, as shown in the picuture, 'ExecuteSQL' processor is producing millions of records, and hence NIFI is running out of memory and getting stuck in the queue and becoming slow.I have made all the possible configurations in the config file, but not helpful. Hence i wanted to restrict the file flow as below, to avoid this situation.

I have 'ExecuteSQL' processor, as show in the picture, which takes input from the incoming flowfile. My requirement is to restrict the flow of the files in queue one-by-one. i.e, incoming only one flowfile to 'ExecuteSQL' processor and allowing it to finish the complete process of nifi flow and once it is done, allowing the next file in the incoming queue of 'ExecuteSQL' processor .

Is there a way to do that?

[#2]

Hi @Sneha Vadhi , May I ask you what are values of Concurrent Tasks and Run Schedule for both ExecuteSQL and ReplaceText processors? Also, you have mentioned that ExecuteSQL generates a million records, did you try increasing the heap size of JVM in bootstrap.conf?

[#3]

Hi Sumit,

Concurrent task:1 and Run Schedule: 0 sec. I tried increasing them, and also tried increasing the heap size in bootstrap.conf but nothing helped.

Is there any other way or would "Wait" processor helps in this case, if yes, what should be the configuration of it ?

[#4]

@Sneha Vadhi , can you please share your configuration for the connection between ReplaceText and ExecuteSQL ? Maybe a little tweak with the Back Pressure and Size Threshold might help you slow down the average load on NiFi.

If you know the average size of each flowfile, you can decrease the threshold or backpressure threshold, so that the queue holds less number of flowfiles and after those files are processed, new files are queued.

There is a Wait Processor for NiFi, but according to the documentation here, you'll have to setup a cache service on Couchbase or Redis.