NIFI: Process 1 flowfile every x seconds, ignore/delete other flow files

[#1]

Hi, 

we have a large number of flowfiles received and processed via NiFi.  Post the NiFi processing we then perform/send the data on via http request. The issue we have at the moment, is that the application (externally to NiFi) can not consume the data fast enough when received from NiFi. What I am looking for is, an ability to only take 1 flowfile every 5 seconds and everything in-between delete/remove.  

Have looked at 1 processor that allows for a per flowfile process, however data then starts getting stuck in the queue.  So although processing 1 file every 5 seconds works, its the delete/remove of any other flow files that I need.  

Thanks.

[#2 · Accepted solution]

Hi Joe, You could use ControlRate to set the max number of flowfiles per second/minute that needs to be routed.

In order to periodically clean the incoming queue you can set a Flowfile Expiration on the incoming connection.

Connection config:

ControlRate config:

With this, flowfiles on the connection will be cleared after they've been for more than 30 seconds on the queue. 

[#3]

Thanks for this, I completely mis-understood this processor.  I read it and thought it was only a control out eg. continue to consume, but only send 1 at a time.  Looks like, this is a one in and one out at a time... looks like this will sort out what I need.  Thanks