Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to copy multiple files from SFTP server to HDFS using Nifi

avatar
Expert Contributor

I have multiple files on my SFTP server with different filenames(filenames have date time appended).Now, I am using ListSftp, RouteOnAttribute, FetchSftp and putHdfs processors. But on the FetchSftp processor I have doubt on how to put all the files on the remote SFTP server to HDFS. Is there any option to provide the list of the file names to "Remote File" property of FetchSftp processor configuration?

Thanks!

1 ACCEPTED SOLUTION

avatar

Hi @Kibrom Gebrehiwot,

The ListSFTP processor will list all the files on your remote SFTP server based on your parameters and will generate one flow file with only metadata (attributes) for each listed file. Then this flow file (with no content yet) will be transmitted to your FetchSFTP processor that will use the metadata of the incoming flow file to actually fetch the content of the remote file into the flow file content.

To use the attributes of the incoming flow file, you should configure the remote file property with something like:

${path}/${filename}

You can find an example here:

https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/List_and_Fetch_SFTP_temp...

Hope this helps.

View solution in original post

2 REPLIES 2

avatar

Hi @Kibrom Gebrehiwot,

The ListSFTP processor will list all the files on your remote SFTP server based on your parameters and will generate one flow file with only metadata (attributes) for each listed file. Then this flow file (with no content yet) will be transmitted to your FetchSFTP processor that will use the metadata of the incoming flow file to actually fetch the content of the remote file into the flow file content.

To use the attributes of the incoming flow file, you should configure the remote file property with something like:

${path}/${filename}

You can find an example here:

https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/List_and_Fetch_SFTP_temp...

Hope this helps.

avatar
Expert Contributor

@Pierre Villard Thank you very much dear! You made my day!