Throttling In Apache NIFI

Question

So the use case that I want to implement is some requests will be landing on my NIFI in XML format and then I want to control its rate and then pick the necessary inputs from the XML and then differentiate based on a report number and use rate limit on the the defined report numbers and then further a rate limit on a article number which I pick from the XML and then I call an API and send the response back the initial response now the issue is I am using control rate processor to hold the requests to 5 per second but with hold flowfile config as I cannot fail the request directly but also I want to fail the queueing requests after 5 second and with a specific status code and also I want to log on which of the three control rate failed the request. Any help how this flow can be created? And also monitor how many requests landed in the server per sec how many succeeded and how many failed per second.

MattWho · Answer

@Former Member

You have what sounds like a rather complex use case here with numerous outputs, timing controls, and routing requirements. Control Rate is very basic in nature (allow X FlowFiles to pass every X amount of time) which depending on volume of FlowFiles can lead to a backlog that ends up resulting in most request to fail after your 5 second requirement (including new that end up delayed more the 5 seconds because they are still queued up behind other FlowFiles behind yoru ControlRate.)

Cloudera offers professional Services to its licensed users that can help design and implement complex use cases. Assisting you through the community would require considerable back and forth and exchange of information to include test files, etc.

Thank you,
Matt

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.