Skip to main content
hegdemahendra
hegdemahendra
Gold ⭐️⭐️
August 26, 2024
Question

Why ConsumeAzureEventhub processor consumes more data even though output connection is configured with less ?

Hello Experts,

We have couple of Eventhub consumers running in nifi 1.16.3 version.

The output connection is configured with default backpressure - 10,000 messages and 1GB.

But I see 'ConsumeAzureEventhub' is keep on consuming data even after crossing backpressure.

hegdemahendra_0-1724675131653.png

hegdemahendra_1-1724675158221.png

What is the reason for this behavior and how to fix this?

@mattw 

 

Thanks

Mahendra

 

 

Thanks

Mahendra

    1 reply

    MattWho
    Titanium ⭐️⭐️⭐️
    MattWho
    Titanium ⭐️⭐️⭐️
    September 9, 2024

    @hegdemahendra 

    The FlowFile connection back pressure thresholds are soft limits.  Once one of the configured back pressure thresholds on reached or exceeded, NiFi will not allow the processor feeding that connection to get scheduled to execute again.

    So in your case no back pressure is being applied, so the ConsumeAzureEventHub processor is being allowed to scheduled to execute.  During a single execution it is consuming more events then the threshold settings.  
    What is the batch size set to in your ConsumeAzureEventHub processor?

    Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

    Thank you,
    Matt

     






    Translate
    Terms & ConditionsAccessibility statement