- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Fetch file last one minute ago from the current time using nifi.
- Labels:
-
Apache NiFi
Created 08-23-2017 05:19 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Im throwing multiple csv files on my hdfs every minute using logstash.
I need to get the files from the past minute from the current time.
Im using nifi in this process.
For example right now is 11:30 AM, I need to get ONLY all the files that are saved 1 minute ago or 11:29AM.
What is the best approach here using nifi?
Thank you.
Created on 08-23-2017 09:23 PM - edited 08-17-2019 06:25 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you're using the GetHDFS processor, you can set the Minimum File Age and Maximum File age properties to
1 minute.
You might have to adjust the properties for a little range, maybe minimum 59 secs and maximum to 61 secs
Created on 08-23-2017 09:23 PM - edited 08-17-2019 06:25 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you're using the GetHDFS processor, you can set the Minimum File Age and Maximum File age properties to
1 minute.
You might have to adjust the properties for a little range, maybe minimum 59 secs and maximum to 61 secs
Created 08-30-2017 09:17 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Did you try the above solution?
Created 08-31-2017 03:26 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Wynner, sorry for the late response. I tried it and its working. Thank you so much! But i need to delete the file so it will just fetch 1 time.
Created on 08-31-2017 12:33 PM - edited 08-17-2019 06:25 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The processor can do that, just set the Keep Source File property to false and the file will be deleted.