In Nifi all the processors in the data flow are executing once per 10 secs

[#1]

Hi,

I am observing the below behavior when there huge number of flow files are in processing by the data flow.

All the nifi processors in the data flow are executing once per 10 secs. For example: I have a updateAttribute processor which is configured to perform 1 concurrent task. I see that it is processing 6 records per minutes even though there are thousands of flow files in the queue.

At the start of loading huge data, the flow works fine. It processes thousands of records in just seconds. But I see the above kind of behavior after sometime. I am unable to understand what is happening in the backend of nifi that is causing for this kind of behavior.

Can I get some help to understand/troubleshoot the issue.

Thanks.

[#2]

Can you please verify the back-pressure threshold of previous processor, which means once the threshold is reached the components right before the queue will no longer be triggered to run anymore until the queue goes below the threshold. You can see it is red in color. You can increase the threshold limit.

[#3]

Back pressure threshold is set to infinite. Queues can hold infinite number of flow files for infinite time.

I see processors not running as usual. They are executing only once for 10 secs.

[#4]

@Anil Reddy

Check the following in your flow, Maximum Timer Driven Thread Count. You can find by doing:

Going to the upper right hand corner of the NiFi UI, left click the three parallel lines, and then this menu will pop-up

Then select Controller Settings and this menu will appear

10 is the default setting. If your value is still 10, then you might be seeing thread starvation in your flow. Try increasing that value. Our rule of thumb for this value is 2 to 4 times the number of CPUs on your system.

How many processors are in your flow and what are the run schedules?