Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Change Data Capture in Databases Using Apache Nifi

Solved Go to solution

Change Data Capture in Databases Using Apache Nifi

New Contributor

Dears,

 

I have to two different databases and the data inside them are identical:

  1. MemSql DB
  2. Postgre DB Frequently data changed

In future, I would like to sync the data between those DBs to capture the data when its changed in the main DB which is the PostgreDB and reflect that changes in the memSql DB.

Is there a way using Nifi to capture data changed in the Postgre DB and move the data changed to MemSql DB?

 

I look forward to hearing from you soon.

Regards,

Ahmed

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Change Data Capture in Databases Using Apache Nifi

Expert Contributor

Hi @ahmedalsaidi 

 

Apache Nifi does not have a generic CDC processor as such. One way you can achieve a CDC like approach is to use the QueryDatabaseTable processor. Please take a look at this article on how to use the QueryDatabaseTable processor to do an incremental fetch of new rows in the source database:

 

https://community.cloudera.com/t5/Community-Articles/Incremental-Fetch-in-NiFi-with-QueryDatabaseTab...

 

Alternatively, you use can use a vendor that specializes in CDC.

 

Please accept this answer as a solution if it helps you.

Steve

View solution in original post

1 REPLY 1
Highlighted

Re: Change Data Capture in Databases Using Apache Nifi

Expert Contributor

Hi @ahmedalsaidi 

 

Apache Nifi does not have a generic CDC processor as such. One way you can achieve a CDC like approach is to use the QueryDatabaseTable processor. Please take a look at this article on how to use the QueryDatabaseTable processor to do an incremental fetch of new rows in the source database:

 

https://community.cloudera.com/t5/Community-Articles/Incremental-Fetch-in-NiFi-with-QueryDatabaseTab...

 

Alternatively, you use can use a vendor that specializes in CDC.

 

Please accept this answer as a solution if it helps you.

Steve

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here