- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Kafka connect with Multi Threading
- Labels:
-
Apache Kafka
Created ‎05-08-2017 11:21 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am trying to load data from RDBMS to kafka topic. I am using kafka jdbc connector provided by Confluent. My questions are
1. If I have millions of data in the database, How can I proceed with multi treading option ?
2. Can we use multiple broker in Kafka connect
3. How can we implement security in this offload from RDBMS to Kafka topic?
4. During data offload lets say my server goes down . How it will behave after kafka server restart?
Please help. This is urgent.
Created ‎05-08-2017 05:19 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
First of all, Nifi is far more mature and easier to use and will do the job much better. That being said, let me try to answer your questions.
1. If I have millions of data in the database, How can I proceed with multi treading option ?
Depends on what you mean by multi threading. Are you getting data from one table? In that case use tasks.max in your property file. Check this link, it has description of how to use it.
2. Can we use multiple broker in Kafka connect
If you are getting data from different tables and you submit jdbc connector to a distributed cluster, it will automatically divide the work in number of tasks equal to number of tables across different workers.
3. How can we implement security in this offload from RDBMS to Kafka topic?
Your RDBS security will be implemented by your database and will require authentication from your program as well as authorization of what that user can read. As for Kafka, the security is described in this link here. You can authenticate clients via SSL or Kerberos and authorization of operations. There is nothing special. You should have that working even if you are not doing this and it will be no different.
4. During data offload lets say my server goes down . How it will behave after kafka server restart?
Please take a look at this discussion. This should answer your questions.
