CDC Postgresql Nifi

Nifi- — Tue, 09 Aug 2022 11:24:01 GMT

Hi Team,

I am new to Apache Nifi, can you please help me to implement CDC for Postgresql one table to log table in Nifi.

I am using Nifi - 1.16.x version.

Re: CDC Postgresql Nifi

MattWho — Wed, 10 Aug 2022 18:14:21 GMT

@Nifi-
You'll need to provide more detail around your use case in order to get more specific assistance.

NiFi offers a number of processor components that can be used to ingest from a database:
- ExecuteSQL
- ExecuteSQLRecord
- CaptureChangeMySQL <-- probably what you are looking for

These ExecuteSQL processors will utilize a DBCPConnectionPool controller service for connecting to your specific Database of choice. SQL is what is needs to passed to these processors in order to fetch database table entries.

The following processors are often used to generate the SQL in different ways needed by your use case to do this in an incremental fashion (for example: generating new SQL for new entries only so you are not fetching entire table over and over)
- GenerateTableFetch
- QueryDatabaseTable
- ListDatabaseTable

The CaptureChangeMySQL processor will output FlowFiles for each individual event. You can then construct a dataflow to write these events to your choice of location. That might be some other database.

Once you have your dataflow created for ingesting entries from your table in to NiFi, you'll need to use other processors within your dataflow for any routing or manipulation of that ingested data you may want to do before sending to a processor to write to the desired destination. Possibly using PutDatabaseRecord processor for example?

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

question Re: CDC Postgresql Nifi in Support Questions

CDC Postgresql Nifi

Re: CDC Postgresql Nifi