Support Questions
Find answers, ask questions, and share your expertise

Fetch files from remote server and put it to HDFS

I would like to fetch files from remote server from ../abc/xyz/kpi path and put it to HDFS and the files in the remote path is getting updated everyday. Being a new person to NiFi I would like to understand which are best processors that can help me here.

1 ACCEPTED SOLUTION

Super Guru
@Sai Krishna Makineni

You can use List/Fetch SFTP processors and then Store the Fetched files into HDFS by using PutHDFS processor.

Flow:

1.ListSFTP (or) ListFTP //list the files fro remote path
2.Remote Processor group //distribute the load across the cluster
3.FetchSFTP (or) FetchFTP //fetch the files from remote
4.PutHDFS //store the fetched files into HDFS

Please refer to this link for more details regards to fetching files from remote path and usage of List/Fetch processors in NiFi.

View solution in original post

2 REPLIES 2

Super Guru
@Sai Krishna Makineni

You can use List/Fetch SFTP processors and then Store the Fetched files into HDFS by using PutHDFS processor.

Flow:

1.ListSFTP (or) ListFTP //list the files fro remote path
2.Remote Processor group //distribute the load across the cluster
3.FetchSFTP (or) FetchFTP //fetch the files from remote
4.PutHDFS //store the fetched files into HDFS

Please refer to this link for more details regards to fetching files from remote path and usage of List/Fetch processors in NiFi.

; ;