question Re: Query regarding Back Pressure Threshold limit in Archives of Support Questions (Read Only)

Query regarding Back Pressure Threshold limit

rohit_ravishank — Thu, 20 Apr 2017 08:32:41 GMT

Hi Team,

I have a set of processors running with default Back Pressure Data Size Threshold of 1 GB. Below is the sample flow,

ListFile -> FetchFile -> CustomProcessor -> PutHDFS

The Custom Processor's concurrency is set to 4. I encounter a single file whose data size is 31 GB. I wanted to know how the data will get processed in this case? Will back Pressure get applied ?

Thanks!

Re: Query regarding Back Pressure Threshold limit

Wynner — Thu, 20 Apr 2017 20:52:55 GMT

@Rohit Ravishankar

Yes the file will be processed and while the 31GB file is being processed, back pressure will be applied.

Back pressure does not prevent files larger than the 1GB from being processed. It just tries to limit the size of the queue to 1GB. Back pressure will not be disabled until the queue size is below the 1GB threshold.

Re: Query regarding Back Pressure Threshold limit

MattWho — Thu, 20 Apr 2017 21:11:06 GMT

@Rohit Ravishankar

Back pressure thresholds are soft limits only. Backpressure is only applied once that threshold has been met or exceeded and remains in affect until the threshold falls below the configured value. Back pressure only affects the processor feeding the connection. processors downstream from where back pressure is being applied continue to run as scheduled.

Re: Query regarding Back Pressure Threshold limit

rohit_ravishank — Fri, 21 Apr 2017 11:53:27 GMT

Got it! Thanks for your answers @Wynner and @Matt Clarke.