question Re: How to download files from sftp, with last modified time X hours. in Archives of Support Questions (Read Only)

How to download files from sftp, with last modified time X hours.

kumarv0686 — Fri, 10 Mar 2017 23:29:04 GMT

Current setup is using getSftp --> puSFTP to sync data from sftp server to a remote linux host.

Problem is with getSFTP i cannot define File age i wish to download, though i found getFile has the min max file age option, however nifi doesn't allow on canvas link between getSFTP and getFile.

I was wondering how i can accomplish this, thanks in advance.

Re: How to download files from sftp, with last modified time X hours.

MattWho — Sun, 18 Aug 2019 12:52:21 GMT

@vikash kumar

You can use the ListSFTP processor to list all files on your SFTP server. The ListSFTP processor will create a 0 byte file with the following additional attributes written to it:

Take the success relationship of the ListSFTP and send it to a routeOnAttribute processor.

Use the routeOnAttribute processor to route on those FlowFiles where the attribute "file.lastModifiedTime" date falls within your desired range to a FetchSFTP processor. All other listed files could just be auto-terminated.

Thanks,

Matt

Re: How to download files from sftp, with last modified time X hours.

kumarv0686 — Sat, 11 Mar 2017 02:33:27 GMT

Thanks Matt, i added the processors you recommended, but the Route on attribute processor is compaining about "file.lastmodified validated against '2017-03-08'T'11:00:00Z' is invalid because no Expressions found.

Re: How to download files from sftp, with last modified time X hours.

MattWho — Sat, 11 Mar 2017 02:43:55 GMT

@vikash kumar

Can you share the NiFi expression language routing rule you created in your RouteOnAttribute processor? The rule must evaluate to "true" before a FlowFile will be routed to that relationship.

Re: How to download files from sftp, with last modified time X hours.

kumarv0686 — Sun, 18 Aug 2019 12:52:13 GMT

sure, here they are

Re: How to download files from sftp, with last modified time X hours.

MattWho — Sat, 11 Mar 2017 03:40:55 GMT

@vikash kumar

Are you looking for files where the "file.lastModifiedTime" is exactly 2017-03-08'T'11:00:00Z?

Or are you looking for all files created at that time and newer?

Re: How to download files from sftp, with last modified time X hours.

kumarv0686 — Sat, 11 Mar 2017 03:45:28 GMT

@Matt Clarke i am looking for the files uploaded on the source sftp in last 24 hours, thanks.

Re: How to download files from sftp, with last modified time X hours.

MattWho — Sun, 18 Aug 2019 12:51:59 GMT

@vikash kumar

The RouteOnAttribute processor expects that a NiFi expression Language statement is used. If the the evaluation of that EL statement results in a true, then the FlowFile will be routed to the corresponding property name's relationship.

Here is an example that will route FlowFile where the value associated to "file.lastModifiedTime" on the incoming FlowFile falls within the last 24 hours to the "last24hours" relationship:

Here is the full EL statement so you can copy it:

${file.lastModifiedTime:toDate("yyyy-MM-dd'T'HH:mm:ssZ"):toNumber():ge(${now():minus(86400000)})}

Thank you,

Matt

Re: How to download files from sftp, with last modified time X hours.

MattWho — Mon, 13 Mar 2017 19:00:58 GMT

@vikash kumar

Did you see the addition I made to my answer above that provided you with a working Expression Language statement to handle your routing? If this solution addressed you question, please accept teh answer.

Thank you,

Matt

Re: How to download files from sftp, with last modified time X hours.

thaishidaib — Wed, 10 May 2017 19:09:48 GMT

Hi Matt,

I'm trying to do the same flow, but I need to get the newest file from an FTP server.

I tried to set the time to 600000ms but the flow did not work.

${file.lastModifiedTime :toDate("yyyy-MM-dd'T'HH:mm:ssZ") :toNumber() :ge(${now():minus(21600000)}) }

How can I get the newest file?

Thanks in advance,

Thais

Hi Matt! I'm trying to do the same flow, but I need to get the newest file from an FTP.

Ho can I do that?

I tried to use 600000 (1 minute) but it did not work.

${file.lastModifiedTime :toDate("yyyy-MM-dd'T'HH:mm:ssZ") :toNumber() :ge(${now():minus(6000000)}) }

Can you help?