Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Fetch files from remote server and put it to HDFS

avatar
Contributor

I would like to fetch files from remote server from ../abc/xyz/kpi path and put it to HDFS and the files in the remote path is getting updated everyday. Being a new person to NiFi I would like to understand which are best processors that can help me here.

1 ACCEPTED SOLUTION

avatar
Master Guru
@Sai Krishna Makineni

You can use List/Fetch SFTP processors and then Store the Fetched files into HDFS by using PutHDFS processor.

Flow:

1.ListSFTP (or) ListFTP //list the files fro remote path
2.Remote Processor group //distribute the load across the cluster
3.FetchSFTP (or) FetchFTP //fetch the files from remote
4.PutHDFS //store the fetched files into HDFS

Please refer to this link for more details regards to fetching files from remote path and usage of List/Fetch processors in NiFi.

View solution in original post

2 REPLIES 2

avatar
Master Guru
@Sai Krishna Makineni

You can use List/Fetch SFTP processors and then Store the Fetched files into HDFS by using PutHDFS processor.

Flow:

1.ListSFTP (or) ListFTP //list the files fro remote path
2.Remote Processor group //distribute the load across the cluster
3.FetchSFTP (or) FetchFTP //fetch the files from remote
4.PutHDFS //store the fetched files into HDFS

Please refer to this link for more details regards to fetching files from remote path and usage of List/Fetch processors in NiFi.

avatar