Support Questions
Find answers, ask questions, and share your expertise

Apache Nifi - Trigger/notification for new data on blob storage Azure?

Solved Go to solution

Apache Nifi - Trigger/notification for new data on blob storage Azure?

Explorer

@ApacheNifi 

I have this flow that take files from Azure Blob Storage (with ListAzureBlobStorage processor).

Now.. is there some trigger or notification mechanism that indicates that new files to be processed have been uploaded to the Azure blob??

Thank you so much

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Apache Nifi - Trigger/notification for new data on blob storage Azure?

Expert Contributor

Hello 

 

According to NiFi documentation, everytime the processor performing a listing of blobs, it auto picks up the new data since last run

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.5.0/org.apache.ni...

State management:

Scope Description

CLUSTERAfter performing a listing of blobs, the timestamp of the newest blob is stored. This allows the Processor to list only blobs that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.

View solution in original post

2 REPLIES 2

Re: Apache Nifi - Trigger/notification for new data on blob storage Azure?

Expert Contributor

Hello 

 

According to NiFi documentation, everytime the processor performing a listing of blobs, it auto picks up the new data since last run

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.5.0/org.apache.ni...

State management:

Scope Description

CLUSTERAfter performing a listing of blobs, the timestamp of the newest blob is stored. This allows the Processor to list only blobs that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.

View solution in original post

Re: Apache Nifi - Trigger/notification for new data on blob storage Azure?

Explorer

Hi, thank you.

However, this is the mechanism of every List processor: it keeps track of the files already processed.
Instead, I was wondering if there was a way to start a trigger or a notification to the flow when new ones arrive in the blob.

Thanks