Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why does NiFi processor ListSFTP sometimes retrieve both flowfiles corresponding to a same file ?

Why does NiFi processor ListSFTP sometimes retrieve both flowfiles corresponding to a same file ?

Hi,

I work on a usecase including following operations :

-> list files from a remote SFTP

-> fetch files listed above

-> write files on HDFS.

I sometimes detect a weird behavior of ListSFTP processor that I am unable to explain.

Sometimes, 2 flowfiles are created by the ListSFTP processor corresponding to the same file on SFTP.

Both flowfiles have the same name/location, same ingestion date/time, but different file.lastModifiedTime and file.size attribute values.

I first thought it was because I was listing files while the process of writing has not been finished yet, but a system renaming files only when writing is complete has been implemented on SFTP side, excluding this hypothesis.

Does someone have already encountered this kind of issue ?

Do you think it is a NiFi-side or SFTP-side matter ?

Thanks a lot.

Benjamin

Don't have an account?
Coming from Hortonworks? Activate your account here