Support Questions
Find answers, ask questions, and share your expertise

Delete file after Tailing

New Contributor

Hi,

I have use case where I need to remove the files only after Tailfile processor completes tailing the file.
 
Currently I have , Tailfile --> PutHDFS
then I will list filenames using ListFile--> ExecuteStreamCommand to remove the files from folder 
How do we know Tailfile processor completed tailing the file so that we can delete that file?
I cant use Getfile or Listsftp/fetchsftp to delete or move a file.
Thank you for your response.
1 REPLY 1

Re: Delete file after Tailing

Cloudera Employee

Tailfile stores local/cluster state to check where it has to start tailing again  or till what time it has already tailed the log file  like List* processor, but state information can not be used by other processor in NiFI to make their own work like flow files attribute value.

Best way I could think of you have you think about buffer time/ retention time for log files and with in that retention time tailfile should be able to taill log files and delete the log files after their retention time, something  like whenever   ExecuteStreamCommand  run for delete it will only delete the files which has last modified time older then X number of days, assuming file which are older then X days is successfully tailed by Tailfile.

 

Thank You.