Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi RouteOnAttribute - filter files older than a day after FetchFile

Nifi RouteOnAttribute - filter files older than a day after FetchFile

Contributor

which property to use in RouteOnAttribute to filter files older a day after FetchFile from source?

Thanks!

 

3 REPLIES 3
Highlighted

Re: Nifi RouteOnAttribute - filter files older than a day after FetchFile

Master Guru

@venkatesh_kanch 

The routeOnAttribute processor expects you to add new dynamic properties.  Each of those dynamic properties you add becomes a new relationship on the processor to use for routing.

Each new property's value is expected to be a NiFi Expression Language statement that results in a boolean true or false.  If based on the FlowFile being processed the expression resolves to "true" then the FlowFile will be routed to this new relationship (property name is used as relationship name).  If none of your dynamic properties resolve to true for a given FlowFile, that FlowFile is routed to the pre-existing unmatched relationship.

 

I recommend performing this routing prior to actually fetching the content, if the intent is to drop any of the FlowFiles.  No sense it using network or disk I/O if you have no need for some content.
The ListFile processor create a number of FlowFile attributes on FlowFile including the file.LastModifiedTime.
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.10.0/org.apach...

 

NiFi Expression Language guide is embedded under help accessible vi the NiFi global menu in the upper right corner of the NiFi UI.  It is also available here:
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html

Note:  Above links from Apache NiFi are valid for most current release.  The embedded docs included in your installed release will be most accurate for your release.

 

Hope this helps,

Matt

Re: Nifi RouteOnAttribute - filter files older than a day after FetchFile

Contributor

Hi @MattWho thanks for the update.

I have used listftp , since I don't own the source cant move or delete files .

This is causing files to be repulled if nifi service is restarted.

 

I will try list files , hope we don't have this pattern here .

 

Really appreciate your help. Thanks

Re: Nifi RouteOnAttribute - filter files older than a day after FetchFile

Master Guru

@venkatesh_kanch 

 

This should not be happening unless you are clearing NiFi state between restarts or NiFi is having a state issue.  When the listSFTP processor runs, it will record state so it does not list the same files more than once.

If the Files being written to the SFTP server from which you are listing are not using dot rename as they are being written, it is possible your ListSFTP processor is listing them while they are still being written to.  Since state is based off timestamps, a file still being written to will have its timestamp updated between executions of the listSFTP processor.  This would result in the file being listed again.
If the standard ssh dot rename method is not being used, you may need to "minimum File Age" property in listSFTP to make sure only files whose timestamp has not updated for X amount of time are listed.  This of course depends on your version of NiFi being used since this property was introduced as part of https://issues.apache.org/jira/browse/NIFI-5977

There is also the possibility if you are using NiFi 1.8 you are hitting this known bug:

https://issues.apache.org/jira/browse/NIFI-5849

 

If you found a solution that worked, please tae a moment to accept the solution to resolve this community question.

 

Thank you,

Matt

Don't have an account?
Coming from Hortonworks? Activate your account here