Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Fetch files from remote server and put it to HDFS

Solved Go to solution
Highlighted

Fetch files from remote server and put it to HDFS

New Contributor

I would like to fetch files from remote server from ../abc/xyz/kpi path and put it to HDFS and the files in the remote path is getting updated everyday. Being a new person to NiFi I would like to understand which are best processors that can help me here.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Fetch files from remote server and put it to HDFS

Super Guru
@Sai Krishna Makineni

You can use List/Fetch SFTP processors and then Store the Fetched files into HDFS by using PutHDFS processor.

Flow:

1.ListSFTP (or) ListFTP //list the files fro remote path
2.Remote Processor group //distribute the load across the cluster
3.FetchSFTP (or) FetchFTP //fetch the files from remote
4.PutHDFS //store the fetched files into HDFS

Please refer to this link for more details regards to fetching files from remote path and usage of List/Fetch processors in NiFi.

2 REPLIES 2

Re: Fetch files from remote server and put it to HDFS

Super Guru
@Sai Krishna Makineni

You can use List/Fetch SFTP processors and then Store the Fetched files into HDFS by using PutHDFS processor.

Flow:

1.ListSFTP (or) ListFTP //list the files fro remote path
2.Remote Processor group //distribute the load across the cluster
3.FetchSFTP (or) FetchFTP //fetch the files from remote
4.PutHDFS //store the fetched files into HDFS

Please refer to this link for more details regards to fetching files from remote path and usage of List/Fetch processors in NiFi.

Re: Fetch files from remote server and put it to HDFS