Support Questions
Find answers, ask questions, and share your expertise

Kafka connect with Multi Threading

Kafka connect with Multi Threading

Rising Star

I am trying to load data from RDBMS to kafka topic. I am using kafka jdbc connector provided by Confluent. My questions are

1. If I have millions of data in the database, How can I proceed with multi treading option ?

2. Can we use multiple broker in Kafka connect

3. How can we implement security in this offload from RDBMS to Kafka topic?

4. During data offload lets say my server goes down . How it will behave after kafka server restart?

Please help. This is urgent.

1 REPLY 1
Highlighted

Re: Kafka connect with Multi Threading

Super Guru

@Arkaprova Saha

First of all, Nifi is far more mature and easier to use and will do the job much better. That being said, let me try to answer your questions.

1. If I have millions of data in the database, How can I proceed with multi treading option ?

Depends on what you mean by multi threading. Are you getting data from one table? In that case use tasks.max in your property file. Check this link, it has description of how to use it.

2. Can we use multiple broker in Kafka connect

If you are getting data from different tables and you submit jdbc connector to a distributed cluster, it will automatically divide the work in number of tasks equal to number of tables across different workers.

3. How can we implement security in this offload from RDBMS to Kafka topic?

Your RDBS security will be implemented by your database and will require authentication from your program as well as authorization of what that user can read. As for Kafka, the security is described in this link here. You can authenticate clients via SSL or Kerberos and authorization of operations. There is nothing special. You should have that working even if you are not doing this and it will be no different.

4. During data offload lets say my server goes down . How it will behave after kafka server restart?

Please take a look at this discussion. This should answer your questions.