question Re: Dynamic Initial Max Value on GenerateTableFetch in Support Questions

Dynamic Initial Max Value on GenerateTableFetch

TonyPath — Sat, 26 Aug 2023 11:35:20 GMT

In a GenerateTableFetch i have set as Maximum-value Columns the column updated_at. In addition i have define the property "initial.maxvalue.updated_at" with an hard-code initial value. However i do not want to hardcode manual the initial value but instead grab it dynamically, like performing an SQL statement to get the max updated value from the table. How i could feed such information in "initial.maxvalue.updated_at" ?

Re: Dynamic Initial Max Value on GenerateTableFetch

SAMSAL — Sat, 26 Aug 2023 13:54:17 GMT

Hi @TonyPath ,

I noticed that the added dynamic property initial.maxvalue.<max_column_name> accept expression language (EL) so since the GenerateTableFetch can take incoming relationship maybe you can use ExecuteSQLRecord to fetch the initial max value from sql into json format then store it in a flow file attribute using EvaluateJsonPath processor which you can reference to set the initial.maxvalue.<max_column_name> property value as follows:

ExecuteSQLRecord Config:

EvaluateJsonPath Config:

GenerateTableFetch Config:

If that helps please accept solution.

Thanks

Re: Dynamic Initial Max Value on GenerateTableFetch

TonyPath — Sun, 27 Aug 2023 15:12:15 GMT

Hi @SAMSAL, I haven't tried it yet but your proposal makes sense, but the execute sql record will run constantly and required manual intervention to stop it so to produce only one flow file. However this is another case

Re: Dynamic Initial Max Value on GenerateTableFetch

SAMSAL — Sun, 27 Aug 2023 22:19:45 GMT

How often are you looking to run the GenerateFetchTable ? If its going to be a batch process then you can set up the a schedule on the top processor using processor config -> Scheduling tab and setting the Run Schedule value. By default this value is set to 0 secs which means its continuously running.