@Gaurav Jain This is the exact use case for why GetSFTP was deprecated in favor of listSFTP and FetchSFTP processors. The ListSFTP processor would run on the primary node only. It produces one 0 byte FlowFile for every file in the listing. All these 0 byte FlowFiles are then sent to a RPG for distribution across cluster. The distributed files are then fed to a FetchSFTP processor that will retrieve the content form the SFTP server and insert it in to the FlowFile at that time. This model eliminates the overhead on the Primary node since it does not need to write the content and it reduces network overhead between nodes since their is no content being send in FlowFiles via the RPG. The only issue you are going to run in to is: https://issues.apache.org/jira/browse/NIFI-1202 This issue is addressed in Apache NiFi 1.2.0 which was just released this week. It will also be addressed in HDF 3.0 which will be released soon. You can work around the issue in older versions by setting a small object backpressure threshold on the connection feeding your RPG. Since this backpressure is a soft limit, you need to put a processor between your listSFTP processor and the RPG that only processes FlowFiles one at a time. I recommend RouteOnAttribute (no configuration needed on processor, simply route the one existing "unmatched" relationship to the RPG and set back pressure on that connection). Thanks, Matt
... View more
