question Re: How to get files based on dates in NiFi? in Archives of Support Questions (Read Only)

How to get files based on dates in NiFi?

manikandan — Tue, 14 Jun 2016 12:53:24 GMT

I'm Newer in hadoop and NiFi. I try to list or get files based on date range (From and To dates) and using windows NiFi. can anyone please give me example or sample of date format.

Thanks

Manikandan

Re: How to get files based on dates in NiFi?

rajkumar_singh — Tue, 14 Jun 2016 13:17:29 GMT

@Manikandan Durairaj

you can take the advantage of regular expression with getFile, something like this https://community.hortonworks.com/questions/38120/how-to-get-files-based-on-the-time-stamp-in-nifi.html

Re: How to get files based on dates in NiFi?

manikandan — Tue, 14 Jun 2016 13:43:13 GMT

Yes @

Rajkumar Singh

already have seen this post before but i have doubt on it, like

Is every file need to give date as name (may be I'm wrong) and need to use expression to filter file name(as date)
Please give me example of how to use date in NIFi

I'm newer in NiFi please explain briefly with examples..

Re: How to get files based on dates in NiFi?

pvillard — Tue, 14 Jun 2016 13:53:05 GMT

Hi @Manikandan Durairaj,

If you don't have dates in the name of your files, the only option I see is to use the combination of ListFile and FetchFile processors. The ListFile processor has the two following properties:

Minimum File Age	The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored
Maximum File Age	The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored

It will allow you to get files with the last modification dates in the given interval. By connecting the output of this processor to the FetchFile processor you will get the files just like the GetFile processor does.

If this is not what you are looking for, we need more details about what you want to achieve.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListFile/index.html

Hope this helps.

Re: How to get files based on dates in NiFi?

manikandan — Tue, 14 Jun 2016 14:15:04 GMT

Hi @

Pierre Villard Thanks for your post.

In Maximum File Age can i give only time (sec, min, Hours) or particular date if i tried to give date (${time:toDate("2016/06/08")}) it shows error
if suppose my file name is like date how can i filter using expression language,please show me example processor properties

Thanks

Manikandan

Re: How to get files based on dates in NiFi?

pvillard — Mon, 19 Aug 2019 08:48:29 GMT

Let's say my files contain the date in its name:

/tmp/test/test-20160501.txt
...
/tmp/test/test-20160531.txt
/tmp/test/test-20160601.txt
...
/tmp/test/test-20160614.txt

If I only want to list files for the month of June:

Note: when hovering the "?" symbol next to a property in processor settings you are able to see if the property supports expression language.

Re: How to get files based on dates in NiFi?

manikandan — Tue, 14 Jun 2016 15:57:07 GMT

Thanks @

Pierre Villard

This is what i was expecting... Thanks once again...