Fetch files from remote server and put it to HDFS

[#1]

I would like to fetch files from remote server from ../abc/xyz/kpi path and put it to HDFS and the files in the remote path is getting updated everyday. Being a new person to NiFi I would like to understand which are best processors that can help me here.

[#2 · Accepted solution]

@Sai Krishna Makineni

You can use List/Fetch SFTP processors and then Store the Fetched files into HDFS by using PutHDFS processor.

Flow:

1.ListSFTP (or) ListFTP //list the files fro remote path
2.Remote Processor group //distribute the load across the cluster
3.FetchSFTP (or) FetchFTP //fetch the files from remote
4.PutHDFS //store the fetched files into HDFS

Please refer to this link for more details regards to fetching files from remote path and usage of List/Fetch processors in NiFi.

[#3]

https://www.welookups.com