Support Questions

Find answers, ask questions, and share your expertise

Query regarding Back Pressure Threshold limit

avatar
Contributor

Hi Team,

I have a set of processors running with default Back Pressure Data Size Threshold of 1 GB. Below is the sample flow,

ListFile -> FetchFile -> CustomProcessor -> PutHDFS

The Custom Processor's concurrency is set to 4. I encounter a single file whose data size is 31 GB. I wanted to know how the data will get processed in this case? Will back Pressure get applied ?

Thanks!

1 ACCEPTED SOLUTION

avatar
@Rohit Ravishankar

Yes the file will be processed and while the 31GB file is being processed, back pressure will be applied.

Back pressure does not prevent files larger than the 1GB from being processed. It just tries to limit the size of the queue to 1GB. Back pressure will not be disabled until the queue size is below the 1GB threshold.

View solution in original post

3 REPLIES 3

avatar
@Rohit Ravishankar

Yes the file will be processed and while the 31GB file is being processed, back pressure will be applied.

Back pressure does not prevent files larger than the 1GB from being processed. It just tries to limit the size of the queue to 1GB. Back pressure will not be disabled until the queue size is below the 1GB threshold.

avatar
Master Mentor

@Rohit Ravishankar

Back pressure thresholds are soft limits only. Backpressure is only applied once that threshold has been met or exceeded and remains in affect until the threshold falls below the configured value. Back pressure only affects the processor feeding the connection. processors downstream from where back pressure is being applied continue to run as scheduled.

avatar
Contributor

Got it! Thanks for your answers @Wynner and @Matt Clarke.