Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

ConsumeAzureEventhub not receiving events after nifi restart

Explorer

Hi !
I was able to setup ConsumeAzureEventhub, and seems to work fine after a fresh configuration

We have an azure eventhub with 4 partitions. 

The consumer is a nifi 2 node cluster, on which ConsumeAzureEventhub processor is set up to run on "primary node"

After the initial setup with a fresh "Storage Container Name" all seems to work fine. 

All new events are processed, and if I manual start/stop the processor it works ok without repeating events, so connection to storage checkpoint seems to work as expected.

 

However after several days, if nifi is restarted or primary node rotated, sometimes the ConsumeAzureEventhub do not reads new events from the queue.
The only workaround we found is setting up a new container (or delete contents of the existing one) which it solves the initial problem but we get a set of events again (repeated)

 

We are using nifi version 1.11.4

The current configuration looks like the attached screenshot

Screen Shot 2020-12-29 at 11.37.30.png

I would appreciate any help with this setup
Thanks in advance!
 

 

 

4 REPLIES 4

Super Guru

check for errors in the logs.

 

stop the processor and make sure it's really stopped.   check the offsets in storagte

 

perhaps the offsets in storage are corrupt or wrong or updated by someone/something else

 

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.12.1/org.apache.n...

New Contributor

Hi, did you solved this issue? we are facing the same problem and not found any solution.

I apprecite any help, thanks!

 

Jorge

Explorer

No yet. We decided to move out ConsumeAzureEventhub because of this issue

Super Guru

Move all events to Kafka

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.