Support Questions

Find answers, ask questions, and share your expertise

Nifi track files already processed?

avatar
Contributor

I have an NFS read only Share that contains thousands of unique files (all with unique names) which I want to process with Nifi. New files are constantly added to this share by another process. Is there any way to keep track of which files Nifi already processes in a prior run, so I don't process them again? I cannot make any changes to files on files on this NFS share (cannot delete or rename them).

1 ACCEPTED SOLUTION

avatar
Master Guru

The ListFile processor keeps track of files it has already seen and only picks up files where the modified date is newer than the last time the processor ran. ListFile produces a flow file for each path to fetch and is used with FetchFile to actually retrieve the file.

View solution in original post

2 REPLIES 2

avatar
Master Guru

The ListFile processor keeps track of files it has already seen and only picks up files where the modified date is newer than the last time the processor ran. ListFile produces a flow file for each path to fetch and is used with FetchFile to actually retrieve the file.

avatar
Contributor

Thanks Bryan, thats just what I was looking for!