Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to clear Nifi Queue

Highlighted

Unable to clear Nifi Queue

Hi, all!

I have a problem with NIFI Queue.

15890-queue.jpg

about 1400000 flow files were stack and I can't clear them...

I tried to set FlowFile Expiration=1 sec, but it doesn't help....

9 REPLIES 9
Highlighted

Re: Unable to clear Nifi Queue

Master Guru

@Oleksandr Solomko

What version of NiFi are you running?

Are you seeing any Out Of Memory errors in your nifi logs? This could be causing issues with emptying the queue.

The fastest way to clear this specific queue now might be to stop both the SplitAVRO and PublishKafka processors. Add an UpdateAttribute processor to your graph with the success relationship set to auto-terminate. Select the queued connection and drag the blue dot (near the arrow end of the connection) to the update Attribute processor. Start the updateAttribute processor and it will start auto-terminating FlowFiles form this connection in batches.

Thanks,

Matt

Highlighted

Re: Unable to clear Nifi Queue

Master Guru

@Oleksandr Solomko

There must be something else going on in your system.

Are you seeing any WARN or ERROR log messages in your nifi logs?

Did you run out of disk space at any time?

Are you seeing any Out Of Memory (OOM) errors in your nifi logs?

Thanks,

Matt

Highlighted

Re: Unable to clear Nifi Queue

there is no Out Of Memory (OOM) errors.

also we don't see any problems with disk space....

Highlighted

Re: Unable to clear Nifi Queue

Master Guru

@Oleksandr Solomko

Any other ERROR or WARN log messages?

---

Is this a standalone NiFi installation or a multi-node NiFi cluster?

- If cluster, are all these FlowFiles queued on just one node?

---

Is this Nifi secured (HTTPS or HTTP)?

I can't reproduce locally.

Thanks, Matt

Highlighted

Re: Unable to clear Nifi Queue

Any other ERROR or WARN log messages? - No

----------

Is this a standalone NiFi installation or a multi-node NiFi cluster? - Cluster

- If cluster, are all these FlowFiles queued on just one node? - if i try to do "list queue" i get : "The queue has no FlowFiles." So i don't know from wich node ar those flow files.....

-----

http

Highlighted

Re: Unable to clear Nifi Queue

Master Guru

@Oleksandr Solomko

You can see where these files are queued via the "summary" UI:

16296-screen-shot-2017-06-13-at-111337-am.png

Once the Summary UI opens, select the "CONNECTIONS" tab. You can sort on any column by clicking that column. Once you have found the row for your queued connection, click on the "view connection details icon ( 16297-screen-shot-2017-06-13-at-111637-am.png )on the far right side of the row.

This will pop open a new UI that shows queue breakdown per node in cluster. This will help you identify if you are having a cluster wide issue here or it is localized to one specific node.

If it is just one node with all this queued data, you could manually disconnect this node from your cluster. Then go directly to the URL for that disconnected node. See if you can empty the queue then. Check for ERROR or WARN logs specifically in that nodes nifi-app.log, nifi-user.log, and nifi-bootstrap.log.

What OS and Java version are you running also?

Thanks,

Matt

Highlighted

Re: Unable to clear Nifi Queue

It doesn't help.

also

15911-queue.jpg

15912-queue1.jpg

Highlighted

Re: Unable to clear Nifi Queue

NiFi - Version 1.1.0.2.1.1.0-2

Highlighted

Re: Unable to clear Nifi Queue

Rising Star
@Oleksandr Solomko have you changed the default value of the "nifi.queue.swap.threshold" property in nifi.properties? If so, you may be running into NIFI-3897.
Don't have an account?
Coming from Hortonworks? Activate your account here