Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi Drop FlowFiles on Back Pressure

Highlighted

NiFi Drop FlowFiles on Back Pressure

Hello,

Is it possible to prevent the Back Pressure of one processor from affecting others? Such as in the case of duplicating data to a Dev/UAT environment in addition to a prod environment.

For example, given the attached image, how would i prevent 'Processor A' from affecting the 'Main Processor' and 'Processor B' from operating when 'Processor A' queue becomes full?

While their are options in queue settings such as "Back Pressure Object Threshold" and "Back Pressure Data Size Threshold" they are only good for dealing with bursting, processing dips, or brief infrastructure interruptions. Another option "FlowFile Expiration" exists and solve the majority of the concern by flushing older flow files from the queue, but under high volumes one still needs to resort to increasing the queue size overall.

Is there no way to reach the max size of a queue and have NiFi drop older events (or newer) from the queue rather than put Back Pressure on the preceding processors?

Thanks

~Regards

43733-nifi-queue-question.png

1 REPLY 1
Highlighted

Re: NiFi Drop FlowFiles on Back Pressure

@Matthew Gaetano

Based on the flow design presented above, there is no way to have backpressure from one flow path not affect the other flow path. Currently the only way to remove older flow files is to enable FlowFile Expiration. But that is always enables whether backpressure is enabled or not.

You could split the flow paths completely and then the backpressure from one flow wouldn't affect the other. Or you could disable back pressure completely and just use the FlowFile expiration capability to remove/drop off files older than a certain age.

Don't have an account?
Coming from Hortonworks? Activate your account here