Incremetnals Update and Delete Using NiFi

[#1]

Hi Team,

Please help me on the below issue

I am ingesting the Data From Oracle to PostgresSql Using NiFi.

We have huge data and continuously generating from sources, I want to update attributes and delete attributes using NiFi

My Flow is: QueryDatabaseRecord-->UpdateAttribute-->PutDatabaseRecord

• when matched record found on the final table then define which action you need to take either Update (or) Delete
• if the record not matched in the final dataset then insert the record.

Appreciate your help.

Thank you,

[#2 · Accepted solution]

@Deppu

If you are able to identify insert/updates then set statement.type attribute to the flowfile then based on the attribute value PutDatabaseRecord processor will run Update/Insert/Delete Statements.

-

This is kind of hacky way to do this:

If you are not able to identify inserts/updates then split the records into each record individually using SplitRecord processor then run two/three successive PutDatabaseRecord processors.

• You need to make decision how to identify Deletes/Updates. because if insert fails then you can try with either update statement type (or) delete statement type.

Flow:

1. FirstPutDatabaseRecord //insert statementtype use failure connection to next PutDatabaseRecord
2.SecondPutDatabaseRecord //update statementtype,use failure connection to next PutDatabaseRecord.
3.ThirdPutDatabaseRecord //delete statementtype

Note: I'm guessing this flow but rearrange these processors as per your logic.

[#3]

This solution will not work. The failure flow does not go to next putDatabaseRecord processor because the error is exception. Failure flow files will go nowhere