Member since
10-19-2018
3
Posts
0
Kudos Received
0
Solutions
11-07-2018
05:27 PM
@Emma Ixiato - *** Community Forum Tip: Try to avoid starting a new answer in response to an existing answer. Instead use comments to respond to existing answers. There is no guaranteed order to different answer which can make it hard following a discussion. - For performance reason the method used to determine what source files are listed is very simplistic in nature and is based off the accuracy of the timestamps on the source files. - Lets assume within a source directory you have a 1 kb file written with a timstamp accurate down to seconds (2018-11-06 16:49:22) which NiFi lists. While that listing is occurring another file is being written to same directory but has not completed being written and is still a "." (hidden) file which list processor ignores by default. The listSFTP processor records a timestamp of the newest listed source file in state. On next execution of ListSFTP, only files with a timestamp newer then "2018-11-06 16:49:22" would be listed. So possibly that other file that was till being written was completed and renamed (to remove ".") within same second of last listing. NiFi would then exclude it in next listing. Another possibility is the system writing the files to the source SFTP server directories is not updating the LastModified timestamps on the "new" files. This resulting in some "new" source files with older timestamps. - If any of this is the case, perhaps https://jira.apache.org/jira/browse/NIFI-5406 that is included in Apache NIFi 1.8.x will help. - Thank you, Matt
... View more