Is NiFi reporting a lot of Garbage collection going on? Do you have a large number of FlowFiles queued in your dataflow(s)? Does a data provenance query run against your RouteOnAttribute processor show extremely slow processing of FlowFiles or a complete stoppage? Does a restart of NiFi trigger it to process files again? ... View more

@John T If you are using the listSFTP processor before your FetchSFTP processor , it will produce a zero byte flow flowfile for every FlowFile it finds on the target SFTP server. The listSFTP processor has a "File Filter Regex" where you can specify a java regular expression to limit what is returned to just files containing "file123.txt". For example "*file123.txt" The ListSFTP processor also maintains state so that the same files are not listed each time. so only new files containing file123.txt are listed each time it runs. The FetchSFTp processor is designed to return the content of a specific file and insert it as content to the FlowFile that he FetchSFTP processor is running against. Thanks, Matt ... View more

Possibility to run every second or minute. In reality this means run as often as possible using the allowable number of concurrent tasks during the 10th hour of each day. I your case it sounds like it was able to run at least 10 times in that one hour. ... View more

@Raphaël MARY If you are running a NiFi cluster, by default every node in your cluster will be running this getHDFS processor at 10 am each day. This means every node will be getting a copy of the same files and processing them in the same way. If you are running a cluster, considering changing the configuration of your getHDFS processor so it runs on primary node only. ... View more

@Rohit Ravishankar Back pressure thresholds are soft limits only. Backpressure is only applied once that threshold has been met or exceeded and remains in affect until the threshold falls below the configured value. Back pressure only affects the processor feeding the connection. processors downstream from where back pressure is being applied continue to run as scheduled. ... View more

@Dmitro Vasilenko The ConsumeKafka_0_10 processor allows dynamic properties: You can try adding a new dynamic property to your ConsumeKafka_0_10 processor for "max.message.bytes" with a value of 2147483647 to see if that works for you. List of Kafka properties can be found here: http://kafka.apache.org/documentation.html#configuration Thanks, Matt ... View more

@Blake Colson If you found my initial response helpful in answering your question, please accept the answer. ... View more

@Emily Sharpe The templates directory was left around to assist those users who are moving from NiFi 0.x to NiFi 1.x baseline. Since NiFi 0.x placed all generated templates in this directory, you can copy those templates over to the configured directory in NIFi 1.x and NiFi will load them in to the flow.xml.gz file for you on startup. There really is no other use for this property other then the above migration. Thanks, Matt ... View more