https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-NiFi-be-used-to-pipe-data-from-sql-server-CDC-to-Kafka/m-p/142563#M35577 <P>The SQL Server Enterprise CDC feature can be directed to write data to a table in the SQL server, effectively keeping an audit log in a SQL table. This can be read using the NiFi QueryDatabaseTable processor. It effectively works a bit like a trigger writing and audit table which you can then read incrementally with NiFi. So, the easy answer is yes. You would just create a flow with QueryDatabaseTable (based on the timestamp as max column) -&gt; PutKafka. You may need to SplitAvro between the two depending on the batch size you need to get good throughput, but that's another question <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span></P><P>That said, there are other forms of CDC which may be more efficient, but currently require third party products. Using Passive CDC, which reads the transaction logs of the SQL Server, and therefore does not put additional query load on the processor is an option here. Tools such as Attunity Replicate use this approach and write directly to Kafka. <A href="https://www.brighttalk.com/webcast/9573/214447">See this webinar</A> for an example of how these tools work and how you can use NiFi with them. </P> Wed, 27 Jul 2016 16:10:49 GMT sball 2016-07-27T16:10:49Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-NiFi-be-used-to-pipe-data-from-sql-server-CDC-to-Kafka/m-p/142561#M35575 Fri, 22 Jul 2016 13:27:06 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-NiFi-be-used-to-pipe-data-from-sql-server-CDC-to-Kafka/m-p/142561#M35575 surajbirla 2016-07-22T13:27:06Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-NiFi-be-used-to-pipe-data-from-sql-server-CDC-to-Kafka/m-p/142562#M35576 <P>NiFi currently doesn't support CDC from transaction logs, but you can utilize DB triggers (hitting the target table frequently to get the change records) and publish the change db records to kafaka.</P> Mon, 25 Jul 2016 20:37:43 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-NiFi-be-used-to-pipe-data-from-sql-server-CDC-to-Kafka/m-p/142562#M35576 mpandit 2016-07-25T20:37:43Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-NiFi-be-used-to-pipe-data-from-sql-server-CDC-to-Kafka/m-p/142563#M35577 <P>The SQL Server Enterprise CDC feature can be directed to write data to a table in the SQL server, effectively keeping an audit log in a SQL table. This can be read using the NiFi QueryDatabaseTable processor. It effectively works a bit like a trigger writing and audit table which you can then read incrementally with NiFi. So, the easy answer is yes. You would just create a flow with QueryDatabaseTable (based on the timestamp as max column) -&gt; PutKafka. You may need to SplitAvro between the two depending on the batch size you need to get good throughput, but that's another question <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span></P><P>That said, there are other forms of CDC which may be more efficient, but currently require third party products. Using Passive CDC, which reads the transaction logs of the SQL Server, and therefore does not put additional query load on the processor is an option here. Tools such as Attunity Replicate use this approach and write directly to Kafka. <A href="https://www.brighttalk.com/webcast/9573/214447">See this webinar</A> for an example of how these tools work and how you can use NiFi with them. </P> Wed, 27 Jul 2016 16:10:49 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-NiFi-be-used-to-pipe-data-from-sql-server-CDC-to-Kafka/m-p/142563#M35577 sball 2016-07-27T16:10:49Z