Created on 06-01-2017 07:37 AM - edited 08-17-2019 11:54 PM
I have a problem with NIFI Queue.
about 1400000 flow files were stack and I can't clear them...
I tried to set FlowFile Expiration=1 sec, but it doesn't help....
Created 06-01-2017 11:12 AM
What version of NiFi are you running?
Are you seeing any Out Of Memory errors in your nifi logs? This could be causing issues with emptying the queue.
The fastest way to clear this specific queue now might be to stop both the SplitAVRO and PublishKafka processors. Add an UpdateAttribute processor to your graph with the success relationship set to auto-terminate. Select the queued connection and drag the blue dot (near the arrow end of the connection) to the update Attribute processor. Start the updateAttribute processor and it will start auto-terminating FlowFiles form this connection in batches.
Created 06-01-2017 11:45 AM
There must be something else going on in your system.
Are you seeing any WARN or ERROR log messages in your nifi logs?
Did you run out of disk space at any time?
Are you seeing any Out Of Memory (OOM) errors in your nifi logs?
Created 06-01-2017 12:27 PM
there is no Out Of Memory (OOM) errors.
also we don't see any problems with disk space....
Created 06-01-2017 01:13 PM
Any other ERROR or WARN log messages?
Is this a standalone NiFi installation or a multi-node NiFi cluster?
- If cluster, are all these FlowFiles queued on just one node?
Is this Nifi secured (HTTPS or HTTP)?
I can't reproduce locally.
Created 06-02-2017 09:12 AM
Any other ERROR or WARN log messages? - No
Is this a standalone NiFi installation or a multi-node NiFi cluster? - Cluster
- If cluster, are all these FlowFiles queued on just one node? - if i try to do "list queue" i get : "The queue has no FlowFiles." So i don't know from wich node ar those flow files.....
Created on 06-13-2017 03:21 PM - edited 08-17-2019 11:54 PM
You can see where these files are queued via the "summary" UI:
Once the Summary UI opens, select the "CONNECTIONS" tab. You can sort on any column by clicking that column. Once you have found the row for your queued connection, click on the "view connection details icon ( )on the far right side of the row.
This will pop open a new UI that shows queue breakdown per node in cluster. This will help you identify if you are having a cluster wide issue here or it is localized to one specific node.
If it is just one node with all this queued data, you could manually disconnect this node from your cluster. Then go directly to the URL for that disconnected node. See if you can empty the queue then. Check for ERROR or WARN logs specifically in that nodes nifi-app.log, nifi-user.log, and nifi-bootstrap.log.
What OS and Java version are you running also?
Created on 06-01-2017 11:31 AM - edited 08-17-2019 11:54 PM
It doesn't help.
Created 06-01-2017 11:32 AM
NiFi - Version 18.104.22.168.1.1.0-2
Created 06-13-2017 03:25 PM