question Re: Proof whether flowfile content already exists in database - NIFI in Support Questions

Proof whether flowfile content already exists in database - NIFI

code — Fri, 05 Aug 2022 11:16:42 GMT

Hi,

I have one flowfile with data I get from an external source (stock of a supplier). The products are already in a database. So now I have to check whether the stock has changed. If the stock of my flowfile is equal to the one in the database, I don't want to write the entry into the database (avoid duplicates).

Maybe someone of you has an idea how to implement this best.

Many thanks for your help!

Re: Proof whether flowfile content already exists in database - NIFI

SAMSAL — Fri, 05 Aug 2022 13:28:07 GMT

Hi ,

What is the flowfile content format? Is there an Id you can check using the LookupRecord processor to see if the record already exists or not?

Re: Proof whether flowfile content already exists in database - NIFI

code — Mon, 08 Aug 2022 08:15:27 GMT

Hey,

thanks for your answer!

I use UpdateRecordProcessor so the format can be whatever it needs to be 🙂 JSON, CSV, Avro... I don't care.

There is a product id in both flowfiles. But most of the time the id is already in the database but just with another stock. So just checking whether the id already exists or not is not enough.

Thank you!

Re: Proof whether flowfile content already exists in database - NIFI

SAMSAL — Mon, 08 Aug 2022 16:00:07 GMT

The lookupRecrod processor allows you to specify multiple lookup columns in the DatabaseRecordLookupService incase the id is not enough. I had a similar sitution and I usually defer this kind of check to the DB, so my workflow will call stored proc - with the record parameters - that will check if the record exist or not and based on that decide wither to update or insert. This way Im making only one connection to the DB.

Re: Proof whether flowfile content already exists in database - NIFI

code — Tue, 09 Aug 2022 07:21:00 GMT

Thanks! That sounds like a good solution.

Is it possible that you send me the configuration of your LookupRecord Processor? 🙂

Re: Proof whether flowfile content already exists in database - NIFI

code — Tue, 09 Aug 2022 07:58:06 GMT

Hello again 🙂 I'm currently struggling with telling the LookupRecord to use multiple columns as key (same in DatabaseRecordLookupService). It would be awesome if you could help me with this!

Many thanks!

Re: Proof whether flowfile content already exists in database - NIFI

Abhiram-4455 — Sun, 15 Oct 2023 10:52:46 GMT

Hi i am wokring on it can you share me the flow file configuration and sp it will helpful to me