Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why does ListSFTP retrieve 2 flowfiles for only one file from SFTP ?

Highlighted

Why does ListSFTP retrieve 2 flowfiles for only one file from SFTP ?

Hi,

I work on a usecase including following operations :

-> list files from a remote SFTP

-> fetch files listed above

-> write files on HDFS.

I sometimes detect a weird behavior of ListSFTP processor that I am unable to explain.

Sometimes, 2 flowfiles are created by the ListSFTP processor corresponding to the same file on SFTP.

Both flowfiles have the same name/location, same ingestion date/time, but different file.lastModifiedTime and file.size attribute values.

I first thought it was because I was listing files while the process of writing has not been finished yet, but a system renaming files only when writing is complete has been implemented on SFTP side, excluding this hypothesis.

Does someone have already encountered this kind of issue ?

Do you think it is a NiFi-side or SFTP-side matter ?

Thanks a lot.

Benjamin

Don't have an account?
Coming from Hortonworks? Activate your account here