How to clear MiniFi queues?

[#1]

So, I have a situation where I am using MiniFi and as we know there is no user interface for MiniFi.

I think the queues are backed up in the flows on there but theres no way for me to tell and if it is, I cannot figure out how to clear them?

Can anyone give me any help?

[#2 · Accepted solution]

The best option to inquire about the state of the flow would be to make use of the FlowStatus querying functionality. You can use this to interpret the flow. This is roughly analogous to the statistics available in the NiFi UI. Considerations and talk to build upon this, especially under the context of C2, are definitely important ideas for handling operational ease and understanding of how instances are behaving. Hopefully the flow status querying is helpful in the interim until a more feature rich mechanism is in place.

Issuing a manual flushing of the queue is something that is not likely to be possible so things like backpressure and expiration periods become very important for connections to help mitigate against such issues.

[#3]

Can you provide a link about "Flow status querying"? I can't seem to find anything on it.

[#4]

Hi @Eric Lloyd

Its in the administrator's guide, you will use the minifi.sh script.

https://nifi.apache.org/minifi/system-admin-guide.html

Scroll down to the section on Status Reporting and Querying

[#5]

Definitely thought I had the link on there before. Specifically, you can find it here: https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.m...

[#6]

I have a question regarding your solution of using backpressure and expiration with MiniFi... this seems like a terrible solution because then you lose the data. We cannot just throw away our data here if it gets stuck in a queue so how do you suggest handling this with MiniFi?

[#7]

Just my Two Cents on it,

If you are experiencing a lot of queued flow-files and you want to clear them without having to reload the config.yml, a very quick and dirty way would be to clear all the repos on the minifi edge.

Here how i do it - assuming your minifi sits in /opt

- you will lose all queued data by doing this, but you no longer have queues.(not ideal)

-- before cleanup 
/opt/minifi/bin/minifi.sh flowStatus connection:all:health,stats | tail -2 | head -1| jq '.'


/opt/minifi/bin/minifi.sh stop
rm -rf /opt/minifi/flowfile_repository/*
rm -rf /opt/minifi/content_repository/*
rm -rf /opt/minifi/provenance_repository/*
/opt/minifi/bin/minifi.sh start


-- after cleanup
/opt/minifi/bin/minifi.sh flowStatus connection:all:health,stats | tail -2 | head -1| jq '.'

Ideal solution:

Use the minifi.sh flowStatus cmd to push data into the NiFi server and monitor this types of stuff. From here you can develop nifi flows to act on what needs to be done to those edge minifi. you can even build a dashboard from this data and actually see how to edges are doing.

No doubt Horton has this on the to do list.