question Re: Apache Nifi - Trigger/notification for new data on blob storage Azure? in Support Questions

Apache Nifi - Trigger/notification for new data on blob storage Azure?

Lallagreta — Fri, 09 Apr 2021 15:18:35 GMT

I have this flow that take files from Azure Blob Storage (with ListAzureBlobStorage processor).

Now.. is there some trigger or notification mechanism that indicates that new files to be processed have been uploaded to the Azure blob??

Thank you so much

Re: Apache Nifi - Trigger/notification for new data on blob storage Azure?

Daming Xue — Sat, 10 Apr 2021 01:34:15 GMT

Hello

According to NiFi documentation, everytime the processor performing a listing of blobs, it auto picks up the new data since last run

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.5.0/org.apache.nifi.processors.azure.storage.ListAzureBlobStorage/index.html

State management:

Scope Description

CLUSTER

After performing a listing of blobs, the timestamp of the newest blob is stored. This allows the Processor to list only blobs that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.

Re: Apache Nifi - Trigger/notification for new data on blob storage Azure?

Lallagreta — Mon, 12 Apr 2021 07:15:02 GMT

Hi, thank you.

However, this is the mechanism of every List processor: it keeps track of the files already processed.
Instead, I was wondering if there was a way to start a trigger or a notification to the flow when new ones arrive in the blob.

Thanks