Query regarding Back Pressure Threshold limit

[#1]

Hi Team,

I have a set of processors running with default Back Pressure Data Size Threshold of 1 GB. Below is the sample flow,

ListFile -> FetchFile -> CustomProcessor -> PutHDFS

The Custom Processor's concurrency is set to 4. I encounter a single file whose data size is 31 GB. I wanted to know how the data will get processed in this case? Will back Pressure get applied ?

Thanks!

[#2 · Accepted solution]

@Rohit Ravishankar

Yes the file will be processed and while the 31GB file is being processed, back pressure will be applied.

Back pressure does not prevent files larger than the 1GB from being processed. It just tries to limit the size of the queue to 1GB. Back pressure will not be disabled until the queue size is below the 1GB threshold.

[#3]

@Rohit Ravishankar

Back pressure thresholds are soft limits only. Backpressure is only applied once that threshold has been met or exceeded and remains in affect until the threshold falls below the configured value. Back pressure only affects the processor feeding the connection. processors downstream from where back pressure is being applied continue to run as scheduled.

[#4]

Got it! Thanks for your answers @Wynner and @Matt Clarke.