Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Fetch file last one minute ago from the current time using nifi.

Solved Go to solution
Highlighted

Fetch file last one minute ago from the current time using nifi.

Hi,

Im throwing multiple csv files on my hdfs every minute using logstash.

I need to get the files from the past minute from the current time.

Im using nifi in this process.

For example right now is 11:30 AM, I need to get ONLY all the files that are saved 1 minute ago or 11:29AM.

What is the best approach here using nifi?

Thank you.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Fetch file last one minute ago from the current time using nifi.

@regie canada

If you're using the GetHDFS processor, you can set the Minimum File Age and Maximum File age properties to

1 minute.

You might have to adjust the properties for a little range, maybe minimum 59 secs and maximum to 61 secs

34431-screen-shot-2017-08-23-at-52358-pm.png

View solution in original post

4 REPLIES 4
Highlighted

Re: Fetch file last one minute ago from the current time using nifi.

@regie canada

If you're using the GetHDFS processor, you can set the Minimum File Age and Maximum File age properties to

1 minute.

You might have to adjust the properties for a little range, maybe minimum 59 secs and maximum to 61 secs

34431-screen-shot-2017-08-23-at-52358-pm.png

View solution in original post

Highlighted

Re: Fetch file last one minute ago from the current time using nifi.

@regie canada

Did you try the above solution?

Highlighted

Re: Fetch file last one minute ago from the current time using nifi.

Hi @Wynner, sorry for the late response. I tried it and its working. Thank you so much! But i need to delete the file so it will just fetch 1 time.

Highlighted

Re: Fetch file last one minute ago from the current time using nifi.

@regie canada

The processor can do that, just set the Keep Source File property to false and the file will be deleted.

34613-screen-shot-2017-08-31-at-83255-am.png

Don't have an account?
Coming from Hortonworks? Activate your account here