Support Questions

Find answers, ask questions, and share your expertise

Incremetnals Update and Delete Using NiFi

avatar

Hi Team,

Please help me on the below issue

I am ingesting the Data From Oracle to PostgresSql Using NiFi.

We have huge data and continuously generating from sources, I want to update attributes and delete attributes using NiFi

My Flow is: QueryDatabaseRecord-->UpdateAttribute-->PutDatabaseRecord

  • when matched record found on the final table then define which action you need to take either Update (or) Delete
  • if the record not matched in the final dataset then insert the record.

Appreciate your help.

Thank you,

1 ACCEPTED SOLUTION

avatar
Master Guru

@Deppu

If you are able to identify insert/updates then set statement.type attribute to the flowfile then based on the attribute value PutDatabaseRecord processor will run Update/Insert/Delete Statements.

-

This is kind of hacky way to do this:

If you are not able to identify inserts/updates then split the records into each record individually using SplitRecord processor then run two/three successive PutDatabaseRecord processors.

  • You need to make decision how to identify Deletes/Updates. because if insert fails then you can try with either update statement type (or) delete statement type.

Flow:

1. FirstPutDatabaseRecord //insert statementtype use failure connection to next PutDatabaseRecord
2.SecondPutDatabaseRecord //update statementtype,use failure connection to next PutDatabaseRecord.
3.ThirdPutDatabaseRecord //delete statementtype

Note: I'm guessing this flow but rearrange these processors as per your logic.

View solution in original post

2 REPLIES 2

avatar
Master Guru

@Deppu

If you are able to identify insert/updates then set statement.type attribute to the flowfile then based on the attribute value PutDatabaseRecord processor will run Update/Insert/Delete Statements.

-

This is kind of hacky way to do this:

If you are not able to identify inserts/updates then split the records into each record individually using SplitRecord processor then run two/three successive PutDatabaseRecord processors.

  • You need to make decision how to identify Deletes/Updates. because if insert fails then you can try with either update statement type (or) delete statement type.

Flow:

1. FirstPutDatabaseRecord //insert statementtype use failure connection to next PutDatabaseRecord
2.SecondPutDatabaseRecord //update statementtype,use failure connection to next PutDatabaseRecord.
3.ThirdPutDatabaseRecord //delete statementtype

Note: I'm guessing this flow but rearrange these processors as per your logic.

avatar
New Contributor

This solution will not work. The failure flow does not go to next putDatabaseRecord processor because the error is exception. Failure flow files will go nowhere