Created 06-14-2016 05:53 AM
I'm Newer in hadoop and NiFi. I try to list or get files based on date range (From and To dates) and using windows NiFi. can anyone please give me example or sample of date format.
Thanks
Manikandan
Created 06-14-2016 06:53 AM
If you don't have dates in the name of your files, the only option I see is to use the combination of ListFile and FetchFile processors. The ListFile processor has the two following properties:
Minimum File Age | The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored | |
Maximum File Age | The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored |
It will allow you to get files with the last modification dates in the given interval. By connecting the output of this processor to the FetchFile processor you will get the files just like the GetFile processor does.
If this is not what you are looking for, we need more details about what you want to achieve.
Hope this helps.
Created 06-14-2016 06:17 AM
you can take the advantage of regular expression with getFile, something like this https://community.hortonworks.com/questions/38120/how-to-get-files-based-on-the-time-stamp-in-nifi.h...
Created 06-14-2016 06:43 AM
Yes @
already have seen this post before but i have doubt on it, like
I'm newer in NiFi please explain briefly with examples..
Created 06-14-2016 06:53 AM
If you don't have dates in the name of your files, the only option I see is to use the combination of ListFile and FetchFile processors. The ListFile processor has the two following properties:
Minimum File Age | The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored | |
Maximum File Age | The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored |
It will allow you to get files with the last modification dates in the given interval. By connecting the output of this processor to the FetchFile processor you will get the files just like the GetFile processor does.
If this is not what you are looking for, we need more details about what you want to achieve.
Hope this helps.
Created 06-14-2016 07:15 AM
Hi @
Pierre Villard Thanks for your post.
Thanks
Manikandan
Created on 06-14-2016 08:19 AM - edited 08-19-2019 01:48 AM
Let's say my files contain the date in its name:
If I only want to list files for the month of June:
Note: when hovering the "?" symbol next to a property in processor settings you are able to see if the property supports expression language.
Created 06-14-2016 08:57 AM