NiFi processor's data gets stuck between two processors

[#1]

Hi all, 

I have been seeing this problem increasingly now that the data gets stuck in between two different processors -- the data being as small as few KBs

what could be the cause of it? The ExecuteSQL contains an SQL insert script. 
Moreover, this behavior is random -- occurs randomly on any processor. Does my machine specs count in this? I see my CPU usage and memory are not choking as well.

Any comments on this will be appreciated. Thanks

[#2]

@Kiranq 

How large is your Max Timer Driven Thread pool set to?

I don't know how many components you have on yoru canvas or how many concurrent tasks you have set on your processors, but if the pool is very small and your CPU load average is not high, you could increase the pool size to see if that helps.  This helps if you have a lot of processors trying to share the tread pool or you have some processors with long running threads holding up threads from the pool.

From the shared screenshot we can see your ExecuteSQL processor has not executed in the previous 5 minutes.

Once "stuck", does the FlowFile ever get processed or is it stuck forever until some manual action is preformed by a user?

Do you see any Out of Memory (OOM) or other ERROR level messages in your nifi-app.log?

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Thanks for replying @MattWho.

Yes, Max Timer Driven Thread pool is set to 10 by default. At one time, around 10+ processors run in a single proccess group.

Moreover, the Flowfile is indefinitely queued until I manually stop the next processor and then start it again. Once I start it, it smoothly migrates the data. I have not checked the logs, I will do so now. 

[#4]

@Kiranq 

If your CPU load average is low relative to the number of cores you have on your Server running NiFi, you could increase the Max timer thread pool.  A typical starting point is 2 to 4 times the number of cores.

Also suggest running diagnostics (./nifi.sh diagnostics) and inspecting that output.
Also check if you have any processors configured with a higher concurrent tasks value.

Thanks,
Matt

[#5]

@MattWho I tried to increase the thread count but the issue still persists. 
I was able to locate one queued file in the logs and I am pasting it under:

this particular flowfile got queued and never processed despite having 30 minutes gone. Its merely a 1.4kb file. 

[#6]

@Kiranq 

Any exceptions (ERROR or WARN) being reported in logs?
Any out of memory exceptions?

Maybe get a diagnostics output while flow is in this state to get some insight on running threads.   (./nifi.sh diagnostics -verbose)

Matt

[#7]

@MattWho No, no WARNINGS OR ERROR in the log files. I dont see any out of memory issues as well. 

However, I have gotten the diagnostics file and I am attaching it in the reply. Can you please look at it and give me any clue to further diagnose this queuing issue?

[#8]

@Kiranq 
Nothing stands out to me as problematic in the diagnostics output that relates to the stick FlowFile. 

• The verbose output indicates just 1 queued FlowFile on the canvas
• The Diagnostics output then tells us that this one FlowFile is queued in the connection with ID: 

  d8e8fe6e-38a5-3fee-8933-2e8c6bf55afa

• The UUID of the queued FlowFile is:

  ce9cbd21-455b-4191-80bc-6598609b71a1

• Details of this One FlowFile:

1. If you clear out the FlowFile, does the very next FlowFile that enters this Process group get stuck? 
2. Do FlowFiles only get stuck in front of this one specific processor and everywhere else FlowFiles are being processed fine?
3. How is this specific process group configured?
4. Can you list the connection and verify the content of this FlowFile looks correct?
5. Have you tried adjusting NiFi's logging to DEBUG to see if any useful Debug output is produced for the issue?

Thanks,
Matt