question Re: Files Duplicating using QueryDatabaseTable and PutFile in Archives of Support Questions (Read Only)

Files Duplicating using QueryDatabaseTable and PutFile

Kelvin_Mitchell — Fri, 16 Sep 2022 11:15:41 GMT

I have a very basic flow that I have put together to test a concept for a presentation. (Btw, relatively new to NiFi). The flow is setup like this: QueryDatabaseTable->SplitAvro->ConvertAvroToJSON->Putfile->LogAttributes

The query is very simple for the Query processor: "select * from test2" which has 8 rows of data. When I fire off the flow, basically it keeps generating files over and over containing the same 8 rows of data in each file.

I know this is probably something very simple to fix, but I just can't seem to find the setting where it will just run once...

I looked around on the boards but did not find anything similar (or entirely possible I maybe missed it).

Does anyone have a suggestion?

Thanks!

♣km

Re: Files Duplicating using QueryDatabaseTable and PutFile

umair_khan — Wed, 15 Mar 2017 22:55:28 GMT

Please verify what have you configured for "Maximum-Value Column" in the QueryDatabaseTable processor. Plus how is your table structured. Nifi needs an increment key to determine the maximum id previously fetched.

Following article will help"

https://community.hortonworks.com/articles/51902/incremental-fetch-in-nifi-with-querydatabasetable.html

Re: Files Duplicating using QueryDatabaseTable and PutFile

Kelvin_Mitchell — Wed, 15 Mar 2017 23:34:29 GMT

Thank you! That was it. I had no values in my Maximum-Value Column.

Re: Files Duplicating using QueryDatabaseTable and PutFile

cstanca — Thu, 16 Mar 2017 01:15:35 GMT

@Kelvin Mitchell

Your QueryTable processor queries continuously your table and generates a FlowFile with each query result. Right-click on QueryTable processor, then Configure and Settings tab and you will see that by default Run Schedule is set to 0 seconds. That means query after query. You can change the interval to more seconds as such a new query will be fired later. Keep in mind that your query can be also smart to pick-up only new records. Is query doing that?

+++

If this helped, please vote/accept best answer.

Re: Files Duplicating using QueryDatabaseTable and PutFile

Kelvin_Mitchell — Fri, 17 Mar 2017 20:05:29 GMT

@Constantin Stanca Thank for the follow up. This was very helpful in addition to Umair's answer. I changed the query to pick up the max rowid for the new records. Again, very helpful tip!