Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to get files based on dates in NiFi?

Solved Go to solution
Highlighted

How to get files based on dates in NiFi?

Explorer

I'm Newer in hadoop and NiFi. I try to list or get files based on date range (From and To dates) and using windows NiFi. can anyone please give me example or sample of date format.

Thanks

Manikandan

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to get files based on dates in NiFi?

Hi @Manikandan Durairaj,

If you don't have dates in the name of your files, the only option I see is to use the combination of ListFile and FetchFile processors. The ListFile processor has the two following properties:

Minimum File Age

The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored

Maximum File AgeThe maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored

It will allow you to get files with the last modification dates in the given interval. By connecting the output of this processor to the FetchFile processor you will get the files just like the GetFile processor does.

If this is not what you are looking for, we need more details about what you want to achieve.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListFile/index...

Hope this helps.

View solution in original post

6 REPLIES 6
Highlighted

Re: How to get files based on dates in NiFi?

@Manikandan Durairaj

you can take the advantage of regular expression with getFile, something like this https://community.hortonworks.com/questions/38120/how-to-get-files-based-on-the-time-stamp-in-nifi.h...

Highlighted

Re: How to get files based on dates in NiFi?

Explorer

Yes @

Rajkumar Singh

already have seen this post before but i have doubt on it, like

  • Is every file need to give date as name (may be I'm wrong) and need to use expression to filter file name(as date)
  • Please give me example of how to use date in NIFi

I'm newer in NiFi please explain briefly with examples..

Highlighted

Re: How to get files based on dates in NiFi?

Hi @Manikandan Durairaj,

If you don't have dates in the name of your files, the only option I see is to use the combination of ListFile and FetchFile processors. The ListFile processor has the two following properties:

Minimum File Age

The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored

Maximum File AgeThe maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored

It will allow you to get files with the last modification dates in the given interval. By connecting the output of this processor to the FetchFile processor you will get the files just like the GetFile processor does.

If this is not what you are looking for, we need more details about what you want to achieve.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListFile/index...

Hope this helps.

View solution in original post

Highlighted

Re: How to get files based on dates in NiFi?

Explorer

Hi @

Pierre Villard Thanks for your post.

  • In Maximum File Age can i give only time (sec, min, Hours) or particular date if i tried to give date (${time:toDate("2016/06/08")}) it shows error
  • if suppose my file name is like date how can i filter using expression language,please show me example processor properties

Thanks

Manikandan

Highlighted

Re: How to get files based on dates in NiFi?

Let's say my files contain the date in its name:

  • /tmp/test/test-20160501.txt
  • ...
  • /tmp/test/test-20160531.txt
  • /tmp/test/test-20160601.txt
  • ...
  • /tmp/test/test-20160614.txt

If I only want to list files for the month of June:

4996-screen-shot-2016-06-14-at-101148-am.png

Note: when hovering the "?" symbol next to a property in processor settings you are able to see if the property supports expression language.

Re: How to get files based on dates in NiFi?

Explorer

Thanks @

Pierre Villard

This is what i was expecting... Thanks once again...

Don't have an account?
Coming from Hortonworks? Activate your account here