These comments are spot-on, thanks! Also I'd mention if you want to dynamically customize it with incoming flow files, an alternative is to send your flow into GenerateTableFetch (on the primary node only, so your most upstream processor(s) will need to run on the primary node only). GenerateTableFetch (GTF) is like QueryDatabaseTable (QDT), with the big differences being 1) GTF takes incoming flow files, and 2) QDT executes the SQL it generates internally, where GTF sends the SQL out as flow files so some other processor (ExecuteSQL, e.g.) can execute it. This can be used by sending the SQL output from GTF to a Remote Process Group (RPG) pointed at an Input Port on your same cluster. This RPG -> Input Port pattern is used to distribute the flow files among the nodes in the cluster, rather than every node working on the same data (which leads to data duplication as @Abdelkrim Hadjidj mentions above. Downstream from the Input Port, all nodes are processing their subset of the flow files in parallel, so you can send Input Port -> ExecuteSQL. This flow is basically a parallel, distributed version of what QueryDatabaseTable does on one node.
... View more
