How to get files based on dates in NiFi?

[#1]

I'm Newer in hadoop and NiFi. I try to list or get files based on date range (From and To dates) and using windows NiFi. can anyone please give me example or sample of date format.

Thanks

Manikandan

[#2]

@Manikandan Durairaj

you can take the advantage of regular expression with getFile, something like this https://community.hortonworks.com/questions/38120/how-to-get-files-based-on-the-time-stamp-in-nifi.h...

[#3]

Yes @

Rajkumar Singh

already have seen this post before but i have doubt on it, like

• Is every file need to give date as name (may be I'm wrong) and need to use expression to filter file name(as date)
• Please give me example of how to use date in NIFi

I'm newer in NiFi please explain briefly with examples..

[#4 · Accepted solution]

Hi @Manikandan Durairaj,

If you don't have dates in the name of your files, the only option I see is to use the combination of ListFile and FetchFile processors. The ListFile processor has the two following properties:

Minimum File Age	The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored
Maximum File Age	The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored

It will allow you to get files with the last modification dates in the given interval. By connecting the output of this processor to the FetchFile processor you will get the files just like the GetFile processor does.

If this is not what you are looking for, we need more details about what you want to achieve.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListFile/index...

Hope this helps.

[#5]

Hi @

Pierre Villard Thanks for your post.

• In Maximum File Age can i give only time (sec, min, Hours) or particular date if i tried to give date (${time:toDate("2016/06/08")}) it shows error
• if suppose my file name is like date how can i filter using expression language,please show me example processor properties

Thanks

Manikandan

[#6]

Let's say my files contain the date in its name:

• /tmp/test/test-20160501.txt
• ...
• /tmp/test/test-20160531.txt
• /tmp/test/test-20160601.txt
• ...
• /tmp/test/test-20160614.txt

If I only want to list files for the month of June:

Note: when hovering the "?" symbol next to a property in processor settings you are able to see if the property supports expression language.

[#7]

Thanks @

Pierre Villard

This is what i was expecting... Thanks once again...