how to configure ListSFTP + FetchSFTP

Question

I am trying to collect data with NiFi using ListSFTP+FetchSFTP. But in the configuration ListSFTP works correctly but I have problems with the configuration of FetchSFTP. I tried with ${path}/${filename} still the problem persists,I also used only the data path, but that doesn't work either . the problem also I have a folder of several files.

MattWho · Accepted Answer

@Justee

You definitely can't use "/Home/Data/" in the "Remote File" property of the FetchSFTP processor. This must be a the full path to a specific file being fetched, which means it should be getting that path and filename values from the inbound FlowFile's attributes.

Your ListSFTP (which you said is working) would be producing a FlowFile for each File found at target directory non your SFTP server. Those FlowFiles will get queued on the outbound connection containing the ListSFTP's "success" relationship.

That connection is the inbound connection to the FetchSFTP processor that will connect to same SFTP server and retrieve the content for each of those FlowFiles that were produced by the ListSFTP processor.

The "Remote File" property on the FetchSFTP processor should be using NiFi Expression Language (EL) to dynamically set the path and filename being fetched uniquely for each inbound FlowFile.

${path}/${filename}

NiFi is case sensitive, so make sure you are using all lowercase so it matches the attribute names created on the Source FlowFiles by the ListSFTP processor.

When you say the FetchSFTP processor is "not working", what does that mean?
To which relationship are the inbound FlowFiles getting routed when it does not work?
What exception (ERROR) is logged in the nifi-app.log when it fails to fetch the content for an inbound FlowFile?
Make sure when you configured the username and password on the FetchSFTP you did not accidentally add a leading or trailing whitespace.

Hope this helps,

Matt

MattWho · Answer

@Justee

Sharing the configuration of both your listSFTP and FetchSFTP processor, as well as a listing of the target SFTP directory would be helpful to those who could assist you here.

The listSFTP processor should be configured with only the base path where all the files you want to list exist in the "Remote Path" property. If the files you want to list reside within sub-directories of that base directory, make sure you also se "Search Recursively" to true.

The ListSFTP processor will output a NiFi FlowFile for each file that is listed. Also those produced FlowFiles will have FlowFile attributes set on them with:

So you might want to list those FlowFiles in the NiFi connection queue an inspect what values were set on these properties. Are they correct?

These attributes can then be used in the FetchSFTP processor to facilitate the fetching of the content for each FlowFile listed.

Hope this helps,

Matt

Screen Shot 2021-05-06 at 2.18.36 PM.png

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.