NiFi message when emptying queues: "Waiting for destination component to stop"

[#1]

When emptying queues in NiFi, the following message is presented: "Waiting for destination component to stop". Queues have not emptied after > 10 minutes.

Any insights into details of what is happening under the covers? Is this how to recover https://community.hortonworks.com/questions/71525/unable-to-clear-nifi-queue.html ?

[#2]

@Greg Keys What is your downstream processor component it is waiting for?

[#3]

There are a few, including InvokeHTTP and SplitJson

[#4 · Accepted solution]

I've seen something similar to this before when you have a circular flow failure on a processor. Especially with an InvokeHTTP processor. The bad thought is..."Ah the request failed so let me re-route it back to the InvokeHTTP processor because maybe it work the 2nd, 3rd..nth time". The failure flow files loop back onto the originating processor and you get this type of behavior because the flow files typically get penalized, have long expiration periods and they get re-queued again. This becomes a continuous failure loop. A better practice to handle the failure into a new processor instead and pay attention to the amount of retries, expirations, etc. @Matt Clarke has a great post on this: https://community.hortonworks.com/questions/77336/nifi-best-practices-for-error-handling.html

[#5]

Thank you @ccasano It was due to this error handling design and InvokeHTTP not able to establish a connection.

[#6]

@Greg Keys

The only destination processor component that would affect the emptying of a queue would be the processor that connection is attached to. Which processor type is this connection you are trying to empty attached to? Does this processor show any active threads in the upper right corner?

It may take some thread dump analysis to determine why the particular processor is not releasing its threads, if some configuration issue is not obvious.

If you restart NiFi, you are likely to get stuck in the same state again because this downstream processor is likely to run before you get access to the NiFi UI to stop it before it runs.

There are a couple things you do to get around this:

1. Try setting FlowFile expiration on the connection to "1 sec" this allows the controller to handle the deletion of FlowFiles from the queue for you. This is of course if NiFi will allow you to edit the connection while the downstream component is till running.

2. The more likely successful operation is to shutdown NiFi and change the "autoresume state" NiFi configuration in the nifi.properties file from true to false. On restart all components will come up stopped. This will allow you to right click on the connection in question and empty it. This will also make sure the downstream processor is in a completely stopped state so its configuration can be changed. Don't forget to change autoresume state back to true after making you changes or every time NiFi is restarted everything will come up stopped.

Thanks,

Matt