@Gaurav Jain When you find an answer in Hortonworks Community Connections (HCC) that addresses your question, please accept that answer so that other HCC users know what worked for you. Thank you kindly, Matt ... View more

@Gaurav Jain Here is an article i wrote awhile ago that explains the differences between using GetSFTP processor or List and Fetch SFTP processors: https://community.hortonworks.com/articles/97773/how-to-retrieve-files-from-a-sftp-server-using-nif.html Thanks, Matt ... View more

@Gaurav Jain This is the exact use case for why GetSFTP was deprecated in favor of listSFTP and FetchSFTP processors. The ListSFTP processor would run on the primary node only. It produces one 0 byte FlowFile for every file in the listing. All these 0 byte FlowFiles are then sent to a RPG for distribution across cluster. The distributed files are then fed to a FetchSFTP processor that will retrieve the content form the SFTP server and insert it in to the FlowFile at that time. This model eliminates the overhead on the Primary node since it does not need to write the content and it reduces network overhead between nodes since their is no content being send in FlowFiles via the RPG. The only issue you are going to run in to is: https://issues.apache.org/jira/browse/NIFI-1202 This issue is addressed in Apache NiFi 1.2.0 which was just released this week. It will also be addressed in HDF 3.0 which will be released soon. You can work around the issue in older versions by setting a small object backpressure threshold on the connection feeding your RPG. Since this backpressure is a soft limit, you need to put a processor between your listSFTP processor and the RPG that only processes FlowFiles one at a time. I recommend RouteOnAttribute (no configuration needed on processor, simply route the one existing "unmatched" relationship to the RPG and set back pressure on that connection). Thanks, Matt ... View more

I literally hit the "tab" key on my keyboard. ... View more

@Sertac Kaya Glad you were able to get the performance improvement you were looking for by allowing your NiFi instance access to additional system threads. If this answer helped you get to your solution, please mark it as accepted. Thank you, Matt ... View more

@Gaurav Jain Each node in a cluster is responsible for working on its own FlowFiles. Each node is unaware of what FlowFiles other nodes are working on. If a NiFi processor component is working on a FlowFile at the time the Node goes down, the transformation work will start over once that the node is running again. A node disconnecting will not cause processing of FlowFiles to stop on the disconnected node. Processors that do transformation of FlowFile content will produce a new FlowFile once the transformation is complete. So if failure exists mid processing, the original remains on the incoming queue to the processor and the intermediate work is lost. This is how NiFi ensures no data loss occurs in unexpected failures. That being said Data plane High Availability (HA) is one of NiFi's roadmap items. Thanks, Matt ... View more

@Sertac Kaya FlowFiles are transferred in a batches between process groups, but that transfer amounts to a updated FlowFile records. This transfer should take fractions of a ms to complete. So many threads should execute per second. So this raises the question of whether your flow is thread starved, concurrent tasks have been over allocated across your processors, your NiFi max timer driven thread count is to low, or your disk IO is very high. I would start by looking at your "Max Timer Driven Thread Count" settings. The default is only 10. By default every component you add to the NiFi canvas uses Timer driven threads. The above count restricts how many system thread can be allocated to components at any one time. I setup a simple 4 cpu vm running a default configuration. The number of FlowFiles passed through the connection between process group 1 and process group 2 ranged between 7084/second to 12,200/second. Thanks, Matt ... View more