question Re: How can I run kafka connect to import data from mysql to kafka? in Archives of Support Questions (Read Only)

How can I run kafka connect to import data from mysql to kafka?

s09021995 — Tue, 21 Aug 2018 00:02:11 GMT

I am new to kafka and i am trying to use get data from mysql to kafka broker using debezium mysql connector?I am not able to understand how to run kafka connect in distributed mode to use debezium mysql connector.

Re: How can I run kafka connect to import data from mysql to kafka?

JordanMoore — Tue, 21 Aug 2018 04:39:18 GMT

@Shobhna Dhami

Somewhere under /usr/hdp/current/kafka there is a connect-distributed script.

You run this and provide a connect-distributed.properties file.

Assuming you are running a recent Kafka version (above 0.11.0), In the properties file, you would add a line that includes "plugin.path" that points to a directory containing the extracted package of the debezium connector.

As mentioned in the Debezium documentation

Simply download the connector’s plugin archive, extract the JARs into your Kafka Connect environment, and add the directory with the JARs to Kafka Connect’s classpath. Restart your Kafka Connect process to pick up the new JARs.

Kafka Documentation - http://kafka.apache.org/documentation/#connect

Confluent Documentation - https://docs.confluent.io/current/connect/index.html (note: Confluent is not a "custom version" of Kafka, they just provide a stronger ecosystem around it)

Re: How can I run kafka connect to import data from mysql to kafka?

s09021995 — Tue, 21 Aug 2018 18:35:03 GMT

@Jordan Moore

Thanks for the quick reply.
However, I am currently using Kafka version 0.10.0.2.5 , how can I give Debezium mysql connector path.

I am getting error:

ERROR Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:183)
org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector andwhich name matches io.debezium.connector.mysql.MySqlConnector available connectors are: org.apache.kafka.connect.sink.SinkConnector, 
org.apache.kafka.connect.tools.VerifiableSourceConnector,
org.apache.kafka.connect.file.FileStreamSinkConnector, 
org.apache.kafka.connect.file.FileStreamSourceConnector, 
org.apache.kafka.connect.source.SourceConnector, 
org.apache.kafka.connect.tools.VerifiableSinkConnector,
org.apache.kafka.connect.tools.MockSourceConnector, 
org.apache.kafka.connect.tools.MockConnector, 
org.apache.kafka.connect.tools.MockSinkConnector

Re: How can I run kafka connect to import data from mysql to kafka?

JordanMoore — Wed, 22 Aug 2018 02:53:16 GMT

@Shobhna Dhami After "available connectors" it does not list it, so you have not setup the classpath correctly, as I linked to.

In Kafka 0.10, you need to run

$ export CLASSPATH=/path/to/extracted-debezium-folder/*.jar # Replace with the real address
$ connect-distributed ...  # Start Connect Server

You can also perform a request to the /connector-plugins URL address before sending any configuration to verify the Debezium connector was correctly installed.

Re: How can I run kafka connect to import data from mysql to kafka?

s09021995 — Fri, 24 Aug 2018 17:12:03 GMT

@Jordan Moore Thanks for the help.