question Re: Fetch file last one minute ago from the current time using nifi. in Archives of Support Questions (Read Only)

Fetch file last one minute ago from the current time using nifi.

regie_canada — Wed, 23 Aug 2017 12:19:50 GMT

Hi,

Im throwing multiple csv files on my hdfs every minute using logstash.

I need to get the files from the past minute from the current time.

Im using nifi in this process.

For example right now is 11:30 AM, I need to get ONLY all the files that are saved 1 minute ago or 11:29AM.

What is the best approach here using nifi?

Thank you.

Wynner — Sun, 18 Aug 2019 01:25:25 GMT

If you're using the GetHDFS processor, you can set the Minimum File Age and Maximum File age properties to

1 minute.

You might have to adjust the properties for a little range, maybe minimum 59 secs and maximum to 61 secs

Wynner — Thu, 31 Aug 2017 04:17:28 GMT

Did you try the above solution?

regie_canada — Thu, 31 Aug 2017 10:26:25 GMT

Hi @Wynner, sorry for the late response. I tried it and its working. Thank you so much! But i need to delete the file so it will just fetch 1 time.

Wynner — Sun, 18 Aug 2019 01:25:17 GMT

The processor can do that, just set the Keep Source File property to false and the file will be deleted.