Nifi RouteOnAttribute - filter files older than a day after FetchFile

[#1]

which property to use in RouteOnAttribute to filter files older a day after FetchFile from source?

Thanks!

[#2]

@Former Member 

The routeOnAttribute processor expects you to add new dynamic properties.  Each of those dynamic properties you add becomes a new relationship on the processor to use for routing.

Each new property's value is expected to be a NiFi Expression Language statement that results in a boolean true or false.  If based on the FlowFile being processed the expression resolves to "true" then the FlowFile will be routed to this new relationship (property name is used as relationship name).  If none of your dynamic properties resolve to true for a given FlowFile, that FlowFile is routed to the pre-existing unmatched relationship.

I recommend performing this routing prior to actually fetching the content, if the intent is to drop any of the FlowFiles.  No sense it using network or disk I/O if you have no need for some content.
The ListFile processor create a number of FlowFile attributes on FlowFile including the file.LastModifiedTime.
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.10.0/org.apache.nifi.processors.standard.ListFile/index.html

NiFi Expression Language guide is embedded under help accessible vi the NiFi global menu in the upper right corner of the NiFi UI.  It is also available here:
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html

Note:  Above links from Apache NiFi are valid for most current release.  The embedded docs included in your installed release will be most accurate for your release.

Hope this helps,

Matt

[#3]

Hi @MattWho thanks for the update.

I have used listftp , since I don't own the source cant move or delete files .

This is causing files to be repulled if nifi service is restarted.

I will try list files , hope we don't have this pattern here .

Really appreciate your help. Thanks

[#4]

@Former Member 

This should not be happening unless you are clearing NiFi state between restarts or NiFi is having a state issue.  When the listSFTP processor runs, it will record state so it does not list the same files more than once.

If the Files being written to the SFTP server from which you are listing are not using dot rename as they are being written, it is possible your ListSFTP processor is listing them while they are still being written to.  Since state is based off timestamps, a file still being written to will have its timestamp updated between executions of the listSFTP processor.  This would result in the file being listed again.
If the standard ssh dot rename method is not being used, you may need to "minimum File Age" property in listSFTP to make sure only files whose timestamp has not updated for X amount of time are listed.  This of course depends on your version of NiFi being used since this property was introduced as part of https://issues.apache.org/jira/browse/NIFI-5977

There is also the possibility if you are using NiFi 1.8 you are hitting this known bug:

https://issues.apache.org/jira/browse/NIFI-5849

If you found a solution that worked, please tae a moment to accept the solution to resolve this community question.

Thank you,

Matt