Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi FetchSFTP cannot find file


NiFi FetchSFTP cannot find file


Running HDF 3.1.0, NiFi 1.5.0.

Flow contains, among other things, a ListSFTP, then a FetchSFTP, then some processing, where only after succes I'd like to delete the file from the SFTP, and since I don't have a DeleteSFTP available, I do another FetchSFTP with completetion strategy set to Delete (first fetch has None for completion strategy).

The issue is that this last FetchSFTP (which actually shouldn't fetch, but just delete) often fails with an error that the file is not found, even though the path is valid/present and just used succesfully for the previous FetchSFTP.

ListSFTP is running only on primary node, and the same for the two FetchSFTP.

Sometimes the last FetchFTP succeeds after a retry, but it seems if I don't run the flow very slowly, like through a RateControl, then it starts failing more and more, to the point that even repeated retries to fetch/delete the file fails on that last cleanup step.

Any ideas on what the reason can be?

I don't suspect I need to treat spaces in file names specifically, right? As it works perfectly fine on the previous FetchSFTP, but on second one, reusing the sftp.remote.filename, it then (often) fails.

I know there could be other patterns to try out, like move/rename/delete etc, but I think the above should work too. Please help.

Don't have an account?
Coming from Hortonworks? Activate your account here