Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi - check if the file in sftp is complete

Highlighted

NiFi - check if the file in sftp is complete

New Contributor

Hello,

How can I verify that the sftp file has finished copying? because listSFTP processor is listenning, but some files do not complete because they have many megabytes and the fetchsftp processor captures it before the transfer ends.
Thanks
2 REPLIES 2

Re: NiFi - check if the file in sftp is complete

Master Guru

@Pepelu Rico

Typically file being transferred to a SFTP server are written using a dot "." filename and then renamed to remove the leading dot "." once transfer has completed.

-

The ListSFTP processor by default has property named "Ignore Dotted Files" which should be set to "true" so that files with names that start with a dot are ignored and not listed.

-

While above is typical it is possible that files being written to your SFTP server are not using the standard dot/rename transfer method.

Is there some other unique naming/renaming happening to indicate transfer is complete?

If so, perhaps you could set up a "File Filter Regex" to avoid listing these files still being transferred.

-

As long as timestamp on file being written is being updated as it is being written to, the listSFTP processor will list the same file again. The ListSFTP processor creates a FlowFile Attribute named "file.size". You could compare this attribute with the FlowFile content "fileSize" attribute after the FetchSFTP processor. If they do not match, you could discard this FlowFile and wait for next listing of dame FlowFile to arrive where these values match. This option is not ideal because it means fetching content multiple times until complete file is fetched.

-

Aside from above there really aren't any other options here.

-

Thank you,

Matt

-

If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

Re: NiFi - check if the file in sftp is complete

Master Guru
@Pepelu Rico

In the upcoming Apache NIFi 1.8 release, you may find following new capability will solve your use case issue here:

https://jira.apache.org/jira/browse/NIFI-5406

Thanks,

Matt