Why ConsumeAzureEventhub processor consumes more data even though output connection is configured with less ?

hegdemahendra — Mon, 26 Aug 2024 12:27:33 GMT

Hello Experts,

We have couple of Eventhub consumers running in nifi 1.16.3 version.

The output connection is configured with default backpressure - 10,000 messages and 1GB.

But I see 'ConsumeAzureEventhub' is keep on consuming data even after crossing backpressure.

What is the reason for this behavior and how to fix this?

@mattw

Thanks

Mahendra

Thanks

Mahendra

Re: Why ConsumeAzureEventhub processor consumes more data even though output connection is configured with less ?

MattWho — Mon, 09 Sep 2024 20:52:50 GMT

@hegdemahendra

The FlowFile connection back pressure thresholds are soft limits. Once one of the configured back pressure thresholds on reached or exceeded, NiFi will not allow the processor feeding that connection to get scheduled to execute again.

So in your case no back pressure is being applied, so the ConsumeAzureEventHub processor is being allowed to scheduled to execute. During a single execution it is consuming more events then the threshold settings.
What is the batch size set to in your ConsumeAzureEventHub processor?

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

question Why ConsumeAzureEventhub processor consumes more data even though output connection is configured with less ? in Support Questions

Why ConsumeAzureEventhub processor consumes more data even though output connection is configured with less ?

Re: Why ConsumeAzureEventhub processor consumes more data even though output connection is configured with less ?