question Re: NiFi for file movement in Support Questions

NiFi for file movement

nifier — Thu, 14 Nov 2024 12:09:58 GMT

I'm new to NiFi and exploring my options.

We are planning to use NiFi as a File Movement platform, where we will be transferring thousands of files everyday of sizes ranging upto 50GB without any extraction, transformation or loading.

We just need to pull the file from one location and push to another location using SFTP and/or S3.

Is NiFi a right tool to use to do this activity ?

Thanks

Re: NiFi for file movement

MattWho — Thu, 14 Nov 2024 21:03:46 GMT

@nifier

Apache NiFi can consume data from many different sources including SFTP and S3 using these processors:

In between those processor you can do many things with that data if you ever choose to do set dynamic attribute for unique destination paths/directories, enhance the data, transform, etc.

Keep in mind that NiFi does not consume and send in a single thread. NiFi must consume the content into its own repositories and then read it from there when writing to the destination. Your described size and volume is not an issue. Your bottle neck if any will be with network and disk I/O depending on your hardware.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: NiFi for file movement

nifier — Fri, 15 Nov 2024 17:10:38 GMT

@MattWho Thanks for your reply 👍