Support Questions
Find answers, ask questions, and share your expertise

Apache Nifi: ListAzureBlobStorage doesn't take all the files in Blob (only a few)

Apache Nifi: ListAzureBlobStorage doesn't take all the files in Blob (only a few)

Explorer

Hi @ApacheNifi,

I have implemented this flow in Nifi.

There are two types of files in the blobstorage (a and b).

The ListAzureBlobStorage processor finds only the b files (which are few) on restart.

To solve the problem I always have to delete the processors (List and RouteOnAttribute) and create a new ones with the same old settings. Why does this happen? does anyone have the same problem?

Thank you so much

Schermata 2021-04-09 alle 10.09.47.png

1 REPLY 1

Re: Apache Nifi: ListAzureBlobStorage doesn't take all the files in Blob (only a few)

Expert Contributor

Hello

 

According to the documentation related to the state management, it will only pull the new files compared to last run

 

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.5.0/org.apache.ni...

 

State management:

Scope Description

CLUSTERAfter performing a listing of blobs, the timestamp of the newest blob is stored. This allows the Processor to list only blobs that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.