How to copy multiple files from SFTP server to HDFS using Nifi

[#1]

I have multiple files on my SFTP server with different filenames(filenames have date time appended).Now, I am using ListSftp, RouteOnAttribute, FetchSftp and putHdfs processors. But on the FetchSftp processor I have doubt on how to put all the files on the remote SFTP server to HDFS. Is there any option to provide the list of the file names to "Remote File" property of FetchSftp processor configuration?

Thanks!

[#2 · Accepted solution]

Hi @Kibrom Gebrehiwot,

The ListSFTP processor will list all the files on your remote SFTP server based on your parameters and will generate one flow file with only metadata (attributes) for each listed file. Then this flow file (with no content yet) will be transmitted to your FetchSFTP processor that will use the metadata of the incoming flow file to actually fetch the content of the remote file into the flow file content.

To use the attributes of the incoming flow file, you should configure the remote file property with something like:

${path}/${filename}

You can find an example here:

https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/List_and_Fetch_SFTP_temp...

Hope this helps.

[#3]

@Pierre Villard Thank you very much dear! You made my day!