Skip to main content
littlesea374
littlesea374
Silver ⭐️⭐️
November 14, 2019
Solved

ListSFTP and FetchFTP duplicate files generated

Hi,

We have a process that generates text files in a ftp server. The process generates the files by writing multiple passes and could take from say more than a minute to a few minutes to generate the files.

 

The nifi ListSFTP in this case generates multiple files as we are using tracking timestamps. The file's timestamp is getting updated continuously during the generation process.

 

We dont have control over the existing process that generates the files and has no notification mechanism in place that can tell us when the file generation is complete.

 

 

What would be the best practice to handle this scenario in a List+FetchSFTP flow to avoid duplicate files generated in the target system. 

 

Please advice.

 

Thank you

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
November 14, 2019

@littlesea374 

 

The ListSFTP processor's only mechanism for ignoring files in the listing directory are those marked as hidden (start with . on linux based systems).   The dot rename of file transfer is pretty common with SFTP.

Now if the files are being streamed into the SFTP server by another processes that does not use some form of dot rename or filename change, you would need the new feature added to listSFTP as part of https://issues.apache.org/jira/browse/NIFI-5977

 

This new feature is part of Apache NiFi 1.10 which adds a couple new configuration properties to the listSFTP processor.  Minimum File Age is what you would need to use.  Only files where the last update time is older than this configured value would be listed.

Hope this helps,

Matt

View in thread

4 replies

MattWho
Titanium ⭐️⭐️⭐️
MattWhoAnswer
Titanium ⭐️⭐️⭐️
November 14, 2019

@littlesea374 

 

The ListSFTP processor's only mechanism for ignoring files in the listing directory are those marked as hidden (start with . on linux based systems).   The dot rename of file transfer is pretty common with SFTP.

Now if the files are being streamed into the SFTP server by another processes that does not use some form of dot rename or filename change, you would need the new feature added to listSFTP as part of https://issues.apache.org/jira/browse/NIFI-5977

 

This new feature is part of Apache NiFi 1.10 which adds a couple new configuration properties to the listSFTP processor.  Minimum File Age is what you would need to use.  Only files where the last update time is older than this configured value would be listed.

Hope this helps,

Matt

Translate
littlesea374
littlesea374Author
Silver ⭐️⭐️
November 14, 2019

Thank you @MattWho , in case we are not able to upgrade to 1.10 or backport it to the current version. Would Wait processor be an acceptable direction to pursue?

Translate
MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
November 14, 2019

@littlesea374 

 

The WAIT processor requires a release signal that is typically created using the NOTIFY processor.

So that really will not help here.

Perhaps you could try setting a penalty on each FlowFile.  Penalized FlowFiles are not processed by the follow on processor until the penalty duration has ended.  This can be done using the ExecuteScript processor after listSFTP:

Screen Shot 2019-11-14 at 4.45.27 PM.png

 

You then set length of "Penalty Duration" via the settings tab.  Set penalty high enough to ensure the file writes have completed.  Of course this does introduce some latency.

What this will not help with is listSFTP still listing the same files multiple times.  As data is written, the timestamp written on that source FlowFile updates, which means it will get listed again as if it is a new file.

but the delay here allows full data to be written and then perhaps you can use a detectDuplicate processor to remove duplicates based on filename before you actually fetch the content.

just some thoughts here, but that JIra is probably best path....

 

Matt

Translate
    Terms & ConditionsAccessibility statement