I work on a usecase including following operations :
-> list files from a remote SFTP
-> fetch files listed above
-> write files on HDFS.
I sometimes detect a weird behavior of ListSFTP processor that I am unable to explain.
Sometimes, 2 flowfiles are created by the ListSFTP processor corresponding to the same file on SFTP.
Both flowfiles have the same name/location, same ingestion date/time, but different file.lastModifiedTime and file.size attribute values.
I first thought it was because I was listing files while the process of writing has not been finished yet, but a system renaming files only when writing is complete has been implemented on SFTP side, excluding this hypothesis.
Does someone have already encountered this kind of issue ?
Do you think it is a NiFi-side or SFTP-side matter ?