Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to get files based on dates in NiFi?

avatar
Rising Star

I'm Newer in hadoop and NiFi. I try to list or get files based on date range (From and To dates) and using windows NiFi. can anyone please give me example or sample of date format.

Thanks

Manikandan

1 ACCEPTED SOLUTION

avatar

Hi @Manikandan Durairaj,

If you don't have dates in the name of your files, the only option I see is to use the combination of ListFile and FetchFile processors. The ListFile processor has the two following properties:

Minimum File Age

The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored

Maximum File AgeThe maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored

It will allow you to get files with the last modification dates in the given interval. By connecting the output of this processor to the FetchFile processor you will get the files just like the GetFile processor does.

If this is not what you are looking for, we need more details about what you want to achieve.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListFile/index...

Hope this helps.

View solution in original post

6 REPLIES 6

avatar
Super Guru
@Manikandan Durairaj

you can take the advantage of regular expression with getFile, something like this https://community.hortonworks.com/questions/38120/how-to-get-files-based-on-the-time-stamp-in-nifi.h...

avatar
Rising Star

Yes @

Rajkumar Singh

already have seen this post before but i have doubt on it, like

  • Is every file need to give date as name (may be I'm wrong) and need to use expression to filter file name(as date)
  • Please give me example of how to use date in NIFi

I'm newer in NiFi please explain briefly with examples..

avatar

Hi @Manikandan Durairaj,

If you don't have dates in the name of your files, the only option I see is to use the combination of ListFile and FetchFile processors. The ListFile processor has the two following properties:

Minimum File Age

The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored

Maximum File AgeThe maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored

It will allow you to get files with the last modification dates in the given interval. By connecting the output of this processor to the FetchFile processor you will get the files just like the GetFile processor does.

If this is not what you are looking for, we need more details about what you want to achieve.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListFile/index...

Hope this helps.

avatar
Rising Star

Hi @

Pierre Villard Thanks for your post.

  • In Maximum File Age can i give only time (sec, min, Hours) or particular date if i tried to give date (${time:toDate("2016/06/08")}) it shows error
  • if suppose my file name is like date how can i filter using expression language,please show me example processor properties

Thanks

Manikandan

avatar

Let's say my files contain the date in its name:

  • /tmp/test/test-20160501.txt
  • ...
  • /tmp/test/test-20160531.txt
  • /tmp/test/test-20160601.txt
  • ...
  • /tmp/test/test-20160614.txt

If I only want to list files for the month of June:

4996-screen-shot-2016-06-14-at-101148-am.png

Note: when hovering the "?" symbol next to a property in processor settings you are able to see if the property supports expression language.

avatar
Rising Star

Thanks @

Pierre Villard

This is what i was expecting... Thanks once again...