question Re: How to copy multiple files from SFTP server to HDFS using Nifi in Archives of Support Questions (Read Only)

How to copy multiple files from SFTP server to HDFS using Nifi

kib — Tue, 21 Feb 2017 22:14:51 GMT

I have multiple files on my SFTP server with different filenames(filenames have date time appended).Now, I am using ListSftp, RouteOnAttribute, FetchSftp and putHdfs processors. But on the FetchSftp processor I have doubt on how to put all the files on the remote SFTP server to HDFS. Is there any option to provide the list of the file names to "Remote File" property of FetchSftp processor configuration?

Thanks!

Re: How to copy multiple files from SFTP server to HDFS using Nifi

pvillard — Tue, 21 Feb 2017 22:30:57 GMT

Hi @Kibrom Gebrehiwot,

The ListSFTP processor will list all the files on your remote SFTP server based on your parameters and will generate one flow file with only metadata (attributes) for each listed file. Then this flow file (with no content yet) will be transmitted to your FetchSFTP processor that will use the metadata of the incoming flow file to actually fetch the content of the remote file into the flow file content.

To use the attributes of the incoming flow file, you should configure the remote file property with something like:

${path}/${filename}

You can find an example here:

https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/List_and_Fetch_SFTP_template.xml

Hope this helps.

Re: How to copy multiple files from SFTP server to HDFS using Nifi

kib — Wed, 22 Feb 2017 13:20:54 GMT

@Pierre Villard Thank you very much dear! You made my day!