- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Fetch files from remote server and put it to HDFS
- Labels:
-
Apache NiFi
Created 08-21-2018 09:45 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I would like to fetch files from remote server from ../abc/xyz/kpi path and put it to HDFS and the files in the remote path is getting updated everyday. Being a new person to NiFi I would like to understand which are best processors that can help me here.
Created 08-21-2018 10:23 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can use List/Fetch SFTP processors and then Store the Fetched files into HDFS by using PutHDFS processor.
Flow:
1.ListSFTP (or) ListFTP //list the files fro remote path
2.Remote Processor group //distribute the load across the cluster
3.FetchSFTP (or) FetchFTP //fetch the files from remote
4.PutHDFS //store the fetched files into HDFS
Please refer to this link for more details regards to fetching files from remote path and usage of List/Fetch processors in NiFi.
Created 08-21-2018 10:23 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can use List/Fetch SFTP processors and then Store the Fetched files into HDFS by using PutHDFS processor.
Flow:
1.ListSFTP (or) ListFTP //list the files fro remote path
2.Remote Processor group //distribute the load across the cluster
3.FetchSFTP (or) FetchFTP //fetch the files from remote
4.PutHDFS //store the fetched files into HDFS
Please refer to this link for more details regards to fetching files from remote path and usage of List/Fetch processors in NiFi.
Created 08-23-2018 04:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
