@SandyClouds Since CaptureChangeMysql does not accept incoming flowfiles you would have to create a separate CaptureChangeMysql flow for each database. This would be exhaustive on 100s of manually created flows. I would have to automate the creation of these flows in a normal nifi environment. In some of the CDC demos I am working with now, i am using CDP Public Cloud Data Flow and able to paramaterize my flows and create flow definitions that are very dynamic then deploy them without touching NiFi or NiFi API and only changing the parameters. This would be an interesting approach to solving 100s of data flows. Another approach I use to begin CDC conversations without actually using "CDC" is to use ExecuteSQL which does accept incoming flowfiles. I use this to grab an entire database and all tables, put all the data in kafka, and all the schema in Schema Registry. This enables me to start to build deeper CDC logic around the same data, including using the same kafka topic per table for updates from other systems, then ultimately deciding how to handle deletes. That said, the complications in creating and operating these types of flows are only enabled when you begin to start capturing data. To me CDC is a journey, not a destination, and we can only begin down that path when we start putting the above concepts in place whether that is CaptureChangeMysql or ExecuteSql with deeper logic for handling changes of the source database.
... View more
