question Re: How to avoid NiFi GetFile recoping of file when KeepSourceFile=true and processing the file at target location in Archives of Support Questions (Read Only)

How to avoid NiFi GetFile recoping of file when KeepSourceFile=true and processing the file at target location

deepeshbhatia21 — Tue, 06 Sep 2016 07:21:59 GMT

I am new to nifi.

I am copying file the file using nifi getfile(Keep the source true) and putfile (/stage).And Every minute there is cronjob looks for file written by putfile , process them and send it hdfs. Soon files got moved out of (/stage) directory,Nifi re-copies the same files.

Is there is any way to avoid re-copy of file in nifi.

Re: How to avoid NiFi GetFile recoping of file when KeepSourceFile=true and processing the file at target location

mburgess — Tue, 06 Sep 2016 07:47:12 GMT

Yes, you can use ListFile -> FetchFile. ListFile will keep track of which files it has read, so it will only list those files that it has not seen before. FetchFile will get the contents of the files passed in. Together they work like GetFile except ListFile keeps track of the files it has read.

Re: How to avoid NiFi GetFile recoping of file when KeepSourceFile=true and processing the file at target location

deepeshbhatia21 — Wed, 07 Sep 2016 05:43:29 GMT

Thanks alot Matt for the quick response. I need one more suggestion. Currently application logs get recycled on daily basis. Let's assume Nifi copies the files at particular point in time using ListFile->FetchFile. There will be some more data will get written before next log file creation.

What's the best way to copy/sync up the file that is being appended using nifi. Current version of nifi is 0.6.0.1.2.0.1-1.

Thanks

Deepesh