You could setup ListFile processor to periodically monitor the directory, and send the listings to an ExecuteCommand processor. The command to execute would be ./yourBinary ${path}/${filename}" The path and filename variables come from the flow files produced by ListFile. ... View more

Hello, Here is a presentation with some more details about record processing: https://www.slideshare.net/BryanBende/apache-nifi-record-processing The main reason record processing achieves good performance is because it allows you to keep all your records together in a single flow file, rather than splitting into a million flow files with 1 record per flow file. It sounds like you were already keeping your 1M records together in a single flow file, so if you got out of memory exceptions, the most likely reason is because you may have loaded the entire content of the flow file into memory (or the library you are using may have). For large files you want to stream data in and out so that you never have the whole content in memory, this is how ConvertRecord works... It reads one record from reader, then passes that record to the writer, then repeats the process, so there is never more than 1 record in memory at a time. These are the best examples of Scripted record reader/writer: https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/test/groovy/org/apache/nifi/record/script ... View more

You should be able to do exactly what you showed with ${gender} as long as your incoming flow files have an attribute called gender. You would have to create that attribute before QueryRecord by using UpdateAttribute or by using a processor that extracts something from the content of the flow into an attribute called gender. ... View more

There currently isn't a processor to do this, but there is an open pull request to add one: https://github.com/apache/nifi/pull/2294 ... View more

If you want to use ${id} as your cache entry identifier, then you need a flow file attribute called id. You could use EvaluateJsonPath first with an expression of id = $.id to extract the id from the json into an attribute. ... View more