Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to integrate kafka to pull data from RDBMS

Solved Go to solution

Re: How to integrate kafka to pull data from RDBMS

@Krishna Srinivas

I am sure have seen this

Sqoop to Kakfa - I don't think so or I have not seen this integration.

You can build data ingestion from SqlServer to Kafka as disucssed in the above link.

Re: How to integrate kafka to pull data from RDBMS

Mentor

@Krishna Srinivas

take a look at nifi, you can sqoop into a spooling dir, have Kafka pick up from there on. Spark streaming in nifi already exists and Storm is going to be included soon. Rough idea of your last inquiry

Sqoop incremental into hdfs directory > watch hdfs dir with nifi > putKafka > Stormspark

You can also split to two pipes in nifi and join into one pipe from two

Re: How to integrate kafka to pull data from RDBMS

Mentor

Look at this example Link for spark streaming an this example for Kafka Link @Krishna Srinivas

Re: How to integrate kafka to pull data from RDBMS

New Contributor

@Krishna Srinivas

You can use Streamsets Data Collector, which is open source, to read from JDBC Databases and put to Kafka topics.

Re: How to integrate kafka to pull data from RDBMS

New Contributor

Guys,

I would prefer Kafka only when data is pushing from an external system.

and another place where I use Kafka, pulled data will be used by multiple parties .so that each consumer connects to kafka topic.

when you have control to pull the data then you can go for custom receivers in Spark. pull what you can consume.

which avoids the extra overhead of maintaining Kafka cluster for balancing the load.

Regards,

@Ram.

Re: How to integrate kafka to pull data from RDBMS

New Contributor

Currently we are implementing a POC in which we require to import real time data from RDBMS to Kafka using Attunity..How to implement the same

Re: How to integrate kafka to pull data from RDBMS

New Contributor

Hi all, is there any update on open source CDC tool for SQL Server?

Don't have an account?
Coming from Hortonworks? Activate your account here