Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to copy multiple files from SFTP server to HDFS using Nifi

Solved Go to solution
Highlighted

How to copy multiple files from SFTP server to HDFS using Nifi

Rising Star

I have multiple files on my SFTP server with different filenames(filenames have date time appended).Now, I am using ListSftp, RouteOnAttribute, FetchSftp and putHdfs processors. But on the FetchSftp processor I have doubt on how to put all the files on the remote SFTP server to HDFS. Is there any option to provide the list of the file names to "Remote File" property of FetchSftp processor configuration?

Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to copy multiple files from SFTP server to HDFS using Nifi

Hi @Kibrom Gebrehiwot,

The ListSFTP processor will list all the files on your remote SFTP server based on your parameters and will generate one flow file with only metadata (attributes) for each listed file. Then this flow file (with no content yet) will be transmitted to your FetchSFTP processor that will use the metadata of the incoming flow file to actually fetch the content of the remote file into the flow file content.

To use the attributes of the incoming flow file, you should configure the remote file property with something like:

${path}/${filename}

You can find an example here:

https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/List_and_Fetch_SFTP_temp...

Hope this helps.

View solution in original post

2 REPLIES 2
Highlighted

Re: How to copy multiple files from SFTP server to HDFS using Nifi

Hi @Kibrom Gebrehiwot,

The ListSFTP processor will list all the files on your remote SFTP server based on your parameters and will generate one flow file with only metadata (attributes) for each listed file. Then this flow file (with no content yet) will be transmitted to your FetchSFTP processor that will use the metadata of the incoming flow file to actually fetch the content of the remote file into the flow file content.

To use the attributes of the incoming flow file, you should configure the remote file property with something like:

${path}/${filename}

You can find an example here:

https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/List_and_Fetch_SFTP_temp...

Hope this helps.

View solution in original post

Highlighted

Re: How to copy multiple files from SFTP server to HDFS using Nifi

Rising Star

@Pierre Villard Thank you very much dear! You made my day!

Don't have an account?
Coming from Hortonworks? Activate your account here