NIFI: Process 1 flowfile every x seconds, ignore/delete other flow files

Question

Hi,

we have a large number of flowfiles received and processed via NiFi. Post the NiFi processing we then perform/send the data on via http request. The issue we have at the moment, is that the application (externally to NiFi) can not consume the data fast enough when received from NiFi. What I am looking for is, an ability to only take 1 flowfile every 5 seconds and everything in-between delete/remove.

Have looked at 1 processor that allows for a per flowfile process, however data then starts getting stuck in the queue. So although processing 1 file every 5 seconds works, its the delete/remove of any other flow files that I need.

Thanks.

bbahamondes · Accepted Answer

Hi Joe, You could use ControlRate to set the max number of flowfiles per second/minute that needs to be routed.

In order to periodically clean the incoming queue you can set a Flowfile Expiration on the incoming connection.

Connection config:

ControlRate config:

With this, flowfiles on the connection will be cleared after they've been for more than 30 seconds on the queue.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.