Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Incremetnals Update and Delete Using NiFi

SOLVED Go to solution
Highlighted

Incremetnals Update and Delete Using NiFi

New Contributor

Hi Team,

Please help me on the below issue

I am ingesting the Data From Oracle to PostgresSql Using NiFi.

We have huge data and continuously generating from sources, I want to update attributes and delete attributes using NiFi

My Flow is: QueryDatabaseRecord-->UpdateAttribute-->PutDatabaseRecord

  • when matched record found on the final table then define which action you need to take either Update (or) Delete
  • if the record not matched in the final dataset then insert the record.

Appreciate your help.

Thank you,

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Incremetnals Update and Delete Using NiFi

Super Guru

@Deppu

If you are able to identify insert/updates then set statement.type attribute to the flowfile then based on the attribute value PutDatabaseRecord processor will run Update/Insert/Delete Statements.

-

This is kind of hacky way to do this:

If you are not able to identify inserts/updates then split the records into each record individually using SplitRecord processor then run two/three successive PutDatabaseRecord processors.

  • You need to make decision how to identify Deletes/Updates. because if insert fails then you can try with either update statement type (or) delete statement type.

Flow:

1. FirstPutDatabaseRecord //insert statementtype use failure connection to next PutDatabaseRecord
2.SecondPutDatabaseRecord //update statementtype,use failure connection to next PutDatabaseRecord.
3.ThirdPutDatabaseRecord //delete statementtype

Note: I'm guessing this flow but rearrange these processors as per your logic.

1 REPLY 1

Re: Incremetnals Update and Delete Using NiFi

Super Guru

@Deppu

If you are able to identify insert/updates then set statement.type attribute to the flowfile then based on the attribute value PutDatabaseRecord processor will run Update/Insert/Delete Statements.

-

This is kind of hacky way to do this:

If you are not able to identify inserts/updates then split the records into each record individually using SplitRecord processor then run two/three successive PutDatabaseRecord processors.

  • You need to make decision how to identify Deletes/Updates. because if insert fails then you can try with either update statement type (or) delete statement type.

Flow:

1. FirstPutDatabaseRecord //insert statementtype use failure connection to next PutDatabaseRecord
2.SecondPutDatabaseRecord //update statementtype,use failure connection to next PutDatabaseRecord.
3.ThirdPutDatabaseRecord //delete statementtype

Note: I'm guessing this flow but rearrange these processors as per your logic.