Created 04-20-2017 01:32 AM
Hi Team,
I have a set of processors running with default Back Pressure Data Size Threshold of 1 GB. Below is the sample flow,
ListFile -> FetchFile -> CustomProcessor -> PutHDFS
The Custom Processor's concurrency is set to 4. I encounter a single file whose data size is 31 GB. I wanted to know how the data will get processed in this case? Will back Pressure get applied ?
Thanks!
Created 04-20-2017 01:52 PM
Yes the file will be processed and while the 31GB file is being processed, back pressure will be applied.
Back pressure does not prevent files larger than the 1GB from being processed. It just tries to limit the size of the queue to 1GB. Back pressure will not be disabled until the queue size is below the 1GB threshold.
Created 04-20-2017 01:52 PM
Yes the file will be processed and while the 31GB file is being processed, back pressure will be applied.
Back pressure does not prevent files larger than the 1GB from being processed. It just tries to limit the size of the queue to 1GB. Back pressure will not be disabled until the queue size is below the 1GB threshold.
Created 04-20-2017 02:11 PM
Back pressure thresholds are soft limits only. Backpressure is only applied once that threshold has been met or exceeded and remains in affect until the threshold falls below the configured value. Back pressure only affects the processor feeding the connection. processors downstream from where back pressure is being applied continue to run as scheduled.
Created 04-21-2017 04:53 AM
Got it! Thanks for your answers @Wynner and @Matt Clarke.