Nifi - How to process one file at a time

[#1]

Shu, please, what is the procedure for processing one file at a time? After unpacking, my files get huge and fill my disk. Is there any configuration in the processor or do I need to restructure my flow? Thank you!

[#2]

@Julio Gazeta

i think you need to use Wait-Notify processors for this case and process each flowfile only once.

Refer to this and this links for more details regards to the design/configure NiFi flow for same exact use case.

[#3]

Shu, I read the article https://community.hortonworks.com/articles/191080/trigger-basedserial-data-processing-in-nifi.html but I did not understand how the RouteOnAttribute processor was configured. Apparently the firstFF property was created, however, I did not see the value of this property in the article? Could you please send a simple example of how to process a list of files, one at a time with the wait and notify processors? Thank you!

[#4]

@Julio Gazeta

Please configure RouteOnAttribute processor as shown below,

RouteOnAttribute configs:

Try to run the flow now.

[#5]

Shu, I realized that the Listfile processor did not pick up the same file more than once, even clearing the status. For example, I had processed file1.gz and moved it to another directory, stopped all processors, cleared the status of ListFile and UpdateAttribute, and restarted processing, and even then the ListFile processor did not pick up file1.gz again. Does Nifi store the reference with this file name in some directory of it that I should delete? Thank you!

[#6]

@Julio Gazeta

I don't think NiFi won't store the reference once we clear off the states in the processor.

In your "d:\\tmp\\input" directory have only one file then clear off all states in ListFile processor then

1.start the processor once and then stop the processor and

2.start the ListFile processor again then you are going to list the file from the directory.

-

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.