Back pressure settings not Honored when a Funnel is in the flow

[#1]

Hi,

I have a small flow created to test priority, but while configuring i noticed that sometimes Back Pressure(Object Threshold) set is not honored per connection when I have a funnel in it. Below is a flow I created, every connection except final connection have default Back Pressure Settings(even Concurrent Tasks is default 1), Last connection out of funnel have Object threshold Set to 10.

When I start all the processors togather except the last LogAttribute processor in the flow, I see Back Pressure Threshold is exceeded in at least 2 connections each time I tested. Am I missing something or is there a logical explanation? Attaching My flow back-pressure.xml

Version: [HDF-2.0.1] [NiFi-1.0.0.2.0.1.0-12]

Thanks,

Jobin George

[#2 · Accepted solution]

Hi Jobin,

The backpressure should be treated as a threshold, not an absolute barrier. Due to scheduling semantics and some processors like e.g. UpdateAttribute doing internal micro-batching, the number of objects (or size) can go over the set limit. It will still engage the backpressure and will not turn it off until the level falls below the threshold, but in general you shouldn't worry too much about the number going slightly above. This is merely due to internal implementation details.

[#3]

Hi @Andrew Grande ,

Thanks for the response, I have seen this when threads are set more than one for some processors and these settings worked so far exactly when funnel is not involved. thats why It is confusing.