@ismail patel Backpressure thresholds are soft limits and some processors do batch processing. The listHDFS processor will produce a list of Files from HDFS and produce a single 0 byte FlowFile for each file in that list. It will then commit all those FlowFiles to the success relationship at once. So if back pressure threshold was set to 5, the ListHDFS processor would still dump all FlowFiles on to it. (even if the listing consisted of 1000s of Files). At that point backpressure would be applied and prevent the listHDFS form running again until the queue dropped back below 5, but this is not the behavior you need here. The RouteOnAttribute processor is one of those processors that works on 1 FlowFile at a time. This allows us to more strictly adhere to the back pressure setting of 5 on its unmatched relationship. The fact that I used a RouteOnAttribute processor is not important, any processor that works on FlowFiles one at a time would work. I picked RouteOnAttribute because it operates off of FlowFile Attributes which live in heap memory which makes processing here very fast. Thanks, Matt ... View more

Thanks. Using an external script seems worse in regards to processing time than regex would be and while custom Java processor seems appealing, I don't believe thats the direction we wish to go. ... View more

There is an expression language guide here: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html For your example you should be able to create an UpdateAttribute processor and add a new property like: myfilename = ${filename:substringAfterLast('/')} ... View more

@uttam kumar Where either of the above answers able to resolve your template issue? If you found either helpful, please accept the answer. Thank you, Matt ... View more

@Sertac Kaya Glad you were able to get the performance improvement you were looking for by allowing your NiFi instance access to additional system threads. If this answer helped you get to your solution, please mark it as accepted. Thank you, Matt ... View more

@Gu Gur And just to add a little detail to Matt's comment, from the documentation, Regular Expressions are entered by adding user-defined properties; the name of the property maps to the Attribute Name into which the result will be placed. ... View more

Hi, There is currently no solution other than to swith back to rsa authent. In addition this library won't work if you try to use rsa with custom key length (I tried 4096 and it failed). You have to edit the ssh_config on the sftp server in order to disable ed25519 (or at least set rsa with a higher priority) I invite you to watch the ticket on nifi's JIRA https://issues.apache.org/jira/browse/NIFI-5816 Regards ... View more

@Wynner Atlast !! Yeah!! Thanks a ton! ... View more

@Avish Saha Some processors work in NiFi FlowFiles in batches. In your case one of the FlowFiles is failing to match on the regex which is causing a session roll back of the entire session. Also seeing as how the processor reports 8,000,000+ tasks in the last 5 minutes, it does not look like it is penalizing that one bad FlowFile. While all FlowFiles routed to a "failure" relationship are penalized, this is not true for all processors when a session rollback occurs. ... View more