NiFi processor's data gets stuck between two processors

[#1]

Hi all, 

I have been seeing this problem increasingly now that the data gets stuck in between two different processors -- the data being as small as few KBs

what could be the cause of it? The ExecuteSQL contains an SQL insert script. 
Moreover, this behavior is random -- occurs randomly on any processor. Does my machine specs count in this? I see my CPU usage and memory are not choking as well.

Any comments on this will be appreciated. Thanks

[#2]

@Kiranq 

How large is your Max Timer Driven Thread pool set to?

I don't know how many components you have on yoru canvas or how many concurrent tasks you have set on your processors, but if the pool is very small and your CPU load average is not high, you could increase the pool size to see if that helps.  This helps if you have a lot of processors trying to share the tread pool or you have some processors with long running threads holding up threads from the pool.

From the shared screenshot we can see your ExecuteSQL processor has not executed in the previous 5 minutes.

Once "stuck", does the FlowFile ever get processed or is it stuck forever until some manual action is preformed by a user?

Do you see any Out of Memory (OOM) or other ERROR level messages in your nifi-app.log?

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Thanks for replying @MattWho.

Yes, Max Timer Driven Thread pool is set to 10 by default. At one time, around 10+ processors run in a single proccess group.

Moreover, the Flowfile is indefinitely queued until I manually stop the next processor and then start it again. Once I start it, it smoothly migrates the data. I have not checked the logs, I will do so now. 

[#4]

@Kiranq 

If your CPU load average is low relative to the number of cores you have on your Server running NiFi, you could increase the Max timer thread pool.  A typical starting point is 2 to 4 times the number of cores.

Also suggest running diagnostics (./nifi.sh diagnostics) and inspecting that output.
Also check if you have any processors configured with a higher concurrent tasks value.

Thanks,
Matt