How to avoid NiFi GetFile recoping of file when KeepSourceFile=true and processing the file at target location

[#1]

I am new to nifi.

I am copying file the file using nifi getfile(Keep the source true) and putfile (/stage).And Every minute there is cronjob looks for file written by putfile , process them and send it hdfs. Soon files got moved out of (/stage) directory,Nifi re-copies the same files.

Is there is any way to avoid re-copy of file in nifi.

[#2 · Accepted solution]

Yes, you can use ListFile -> FetchFile. ListFile will keep track of which files it has read, so it will only list those files that it has not seen before. FetchFile will get the contents of the files passed in. Together they work like GetFile except ListFile keeps track of the files it has read.

[#3]

Thanks alot Matt for the quick response. I need one more suggestion. Currently application logs get recycled on daily basis. Let's assume Nifi copies the files at particular point in time using ListFile->FetchFile. There will be some more data will get written before next log file creation.

What's the best way to copy/sync up the file that is being appended using nifi. Current version of nifi is 0.6.0.1.2.0.1-1.

Thanks

Deepesh