- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Query regarding Back Pressure Threshold limit
- Labels:
-
Apache NiFi
Created ‎04-20-2017 01:32 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Team,
I have a set of processors running with default Back Pressure Data Size Threshold of 1 GB. Below is the sample flow,
ListFile -> FetchFile -> CustomProcessor -> PutHDFS
The Custom Processor's concurrency is set to 4. I encounter a single file whose data size is 31 GB. I wanted to know how the data will get processed in this case? Will back Pressure get applied ?
Thanks!
Created ‎04-20-2017 01:52 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes the file will be processed and while the 31GB file is being processed, back pressure will be applied.
Back pressure does not prevent files larger than the 1GB from being processed. It just tries to limit the size of the queue to 1GB. Back pressure will not be disabled until the queue size is below the 1GB threshold.
Created ‎04-20-2017 01:52 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes the file will be processed and while the 31GB file is being processed, back pressure will be applied.
Back pressure does not prevent files larger than the 1GB from being processed. It just tries to limit the size of the queue to 1GB. Back pressure will not be disabled until the queue size is below the 1GB threshold.
Created ‎04-20-2017 02:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Back pressure thresholds are soft limits only. Backpressure is only applied once that threshold has been met or exceeded and remains in affect until the threshold falls below the configured value. Back pressure only affects the processor feeding the connection. processors downstream from where back pressure is being applied continue to run as scheduled.
Created ‎04-21-2017 04:53 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Got it! Thanks for your answers @Wynner and @Matt Clarke.
