question Incremetnals Update and Delete Using NiFi in Support Questions

Incremetnals Update and Delete Using NiFi

dileepkumar_upp — Wed, 16 Jan 2019 00:47:53 GMT

Hi Team,

Please help me on the below issue

I am ingesting the Data From Oracle to PostgresSql Using NiFi.

We have huge data and continuously generating from sources, I want to update attributes and delete attributes using NiFi

My Flow is: QueryDatabaseRecord-->UpdateAttribute-->PutDatabaseRecord

when matched record found on the final table then define which action you need to take either Update (or) Delete
if the record not matched in the final dataset then insert the record.

Appreciate your help.

Thank you,

Re: Incremetnals Update and Delete Using NiFi

Shu_ashu — Wed, 16 Jan 2019 08:18:27 GMT

@Deppu

If you are able to identify insert/updates then set statement.type attribute to the flowfile then based on the attribute value PutDatabaseRecord processor will run Update/Insert/Delete Statements.

This is kind of hacky way to do this:

If you are not able to identify inserts/updates then split the records into each record individually using SplitRecord processor then run two/three successive PutDatabaseRecord processors.

You need to make decision how to identify Deletes/Updates. because if insert fails then you can try with either update statement type (or) delete statement type.

Flow:

1. FirstPutDatabaseRecord //insert statementtype use failure connection to next PutDatabaseRecord
2.SecondPutDatabaseRecord //update statementtype,use failure connection to next PutDatabaseRecord.
3.ThirdPutDatabaseRecord //delete statementtype

Note: I'm guessing this flow but rearrange these processors as per your logic.

Re: Incremetnals Update and Delete Using NiFi

JohnNi — Fri, 12 Feb 2021 08:10:03 GMT

This solution will not work. The failure flow does not go to next putDatabaseRecord processor because the error is exception. Failure flow files will go nowhere