Skip to main content
Lallagreta
Lallagreta
Silver ⭐️⭐️
April 9, 2021
Solved

Apache Nifi - Trigger/notification for new data on blob storage Azure?

@ApacheNifi 

I have this flow that take files from Azure Blob Storage (with ListAzureBlobStorage processor).

Now.. is there some trigger or notification mechanism that indicates that new files to be processed have been uploaded to the Azure blob??

Thank you so much

 

Daming Xue
Daming XueAnswer
Gold ⭐️
April 10, 2021

Hello 

 

According to NiFi documentation, everytime the processor performing a listing of blobs, it auto picks up the new data since last run

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.5.0/org.apache.nifi.processors.azure.storage.ListAzureBlobStorage/index.html

State management:

Scope Description

CLUSTERAfter performing a listing of blobs, the timestamp of the newest blob is stored. This allows the Processor to list only blobs that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.
View in thread

2 replies

Daming Xue
Gold ⭐️
Daming XueAnswer
Gold ⭐️
April 10, 2021

Hello 

 

According to NiFi documentation, everytime the processor performing a listing of blobs, it auto picks up the new data since last run

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.5.0/org.apache.nifi.processors.azure.storage.ListAzureBlobStorage/index.html

State management:

Scope Description

CLUSTERAfter performing a listing of blobs, the timestamp of the newest blob is stored. This allows the Processor to list only blobs that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.
Translate
    Lallagreta
    LallagretaAuthor
    Silver ⭐️⭐️
    April 12, 2021

    Hi, thank you.

    However, this is the mechanism of every List processor: it keeps track of the files already processed.
    Instead, I was wondering if there was a way to start a trigger or a notification to the flow when new ones arrive in the blob.

    Thanks

    Translate
    Terms & ConditionsAccessibility statement