Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to configure ListSFTP + FetchSFTP

avatar
Explorer

I am trying to collect data with NiFi using ListSFTP+FetchSFTP. But in the configuration ListSFTP works correctly but I have problems with the configuration of FetchSFTP. I tried with ${path}/${filename} still the problem persists,I also used only the data path, but that doesn't work either . the problem also I have a folder of several files.

1 ACCEPTED SOLUTION

avatar
Super Mentor

@Justee 

You definitely can't use "/Home/Data/" in the "Remote File" property of the FetchSFTP processor.  This must be a the full path to a specific file being fetched, which means it should be getting that path and filename values from the inbound FlowFile's attributes.

MattWho_0-1620390199833.png

Your ListSFTP (which you said is working) would be producing a FlowFile for each File found at target directory non your SFTP server.  Those FlowFiles will get queued on the outbound connection containing the ListSFTP's "success" relationship.

That connection is the inbound connection to the FetchSFTP processor that will connect to same SFTP server and retrieve the content for each of those FlowFiles that were produced by the ListSFTP processor.

The "Remote File" property on the FetchSFTP processor should be using NiFi Expression Language (EL) to dynamically set the path and filename being fetched uniquely for each inbound FlowFile.

${path}/${filename}

NiFi is case sensitive, so make sure you are using all lowercase so it matches the attribute names created on the Source FlowFiles by the ListSFTP processor.

When you say the FetchSFTP processor is "not working", what does that mean?
To which relationship are the inbound FlowFiles getting routed when it does not work?
What exception (ERROR) is logged in the nifi-app.log when it fails to fetch the content for an inbound FlowFile?
Make sure when you configured the username and password on the FetchSFTP you did not accidentally add a leading or trailing whitespace.

Hope this helps,

Matt

View solution in original post

3 REPLIES 3

avatar
Super Mentor

@Justee 

Sharing the configuration of both your listSFTP and FetchSFTP processor, as well as a listing of the target SFTP directory would be helpful to those who could assist you here. 

The listSFTP processor should be configured with only the base path where all the files you want to list exist  in the "Remote Path" property.  If the files you want to list reside within sub-directories of that base directory, make sure you also se "Search Recursively" to true.

The ListSFTP processor will output a NiFi FlowFile for each file that is listed.  Also those produced FlowFiles will have FlowFile attributes set on them with:

MattWho_0-1620325087873.png

 

So you might want to list those FlowFiles in the NiFi connection queue an inspect what values were set on these properties.  Are they correct?  

These attributes can then be used in the FetchSFTP processor to facilitate the fetching of the content for each FlowFile listed.

Hope this helps,

Matt


Screen Shot 2021-05-06 at 2.18.36 PM.png

avatar
Explorer

Hi@MattWho, I have a folder of files. Here is the configuration I tried to set up but the problem is on remote filename I tried: remote filename = ${/home/Data/}  remote fiename = /home/Data/, but both do not work. 

 

ListSFTP  settingsListSFTP settingsFetchSFTP settingsFetchSFTP settings

avatar
Super Mentor

@Justee 

You definitely can't use "/Home/Data/" in the "Remote File" property of the FetchSFTP processor.  This must be a the full path to a specific file being fetched, which means it should be getting that path and filename values from the inbound FlowFile's attributes.

MattWho_0-1620390199833.png

Your ListSFTP (which you said is working) would be producing a FlowFile for each File found at target directory non your SFTP server.  Those FlowFiles will get queued on the outbound connection containing the ListSFTP's "success" relationship.

That connection is the inbound connection to the FetchSFTP processor that will connect to same SFTP server and retrieve the content for each of those FlowFiles that were produced by the ListSFTP processor.

The "Remote File" property on the FetchSFTP processor should be using NiFi Expression Language (EL) to dynamically set the path and filename being fetched uniquely for each inbound FlowFile.

${path}/${filename}

NiFi is case sensitive, so make sure you are using all lowercase so it matches the attribute names created on the Source FlowFiles by the ListSFTP processor.

When you say the FetchSFTP processor is "not working", what does that mean?
To which relationship are the inbound FlowFiles getting routed when it does not work?
What exception (ERROR) is logged in the nifi-app.log when it fails to fetch the content for an inbound FlowFile?
Make sure when you configured the username and password on the FetchSFTP you did not accidentally add a leading or trailing whitespace.

Hope this helps,

Matt