Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

When to use Kafka Connect vs. Producer and Consumer?

avatar
Contributor
 
1 ACCEPTED SOLUTION

avatar
Super Guru

@Andi Sonde

You will use Kafka clients when you are a developer, you want to connect an application to Kafka and can modify the code of the application, and you want to push data into Kafka or pull data from Kafka.

You will use Connect to connect Kafka to datastores that you did not write and can’t or won’t modify their code. For data stores where a connector already exists, Connect can be used by non-developers who will only need to configure the connectors.

If you need to connect Kafka to a data store and a connector does not exist yet, you can choose between writing an app using the Kafka clients or the Connect APIs. Connect is recommended because it provides out of the box features like configuration management, offset storage, parallelization, error handling, support for different data types and standard management REST API. Writing a small app that connects Kafka to a data store sounds simple, but there are many little details you will need to handle around data types and configuration that make the task non-trivial - Kafka connect handles most of this for you, allowing you to focus on transporting data to and from the external stores.

I realize that is a bit out of scope, but I'd like to recommend Apache NiFi as an alternative to Kafka Connect. See: https://nifi.apache.org/

NiFi is a good alternative to Flume too. NiFi provide an advanced data management capability covering easily Kafka producer or Kafka consumer needs with no coding, visually, at most, some regular expressions.

View solution in original post

1 REPLY 1

avatar
Super Guru

@Andi Sonde

You will use Kafka clients when you are a developer, you want to connect an application to Kafka and can modify the code of the application, and you want to push data into Kafka or pull data from Kafka.

You will use Connect to connect Kafka to datastores that you did not write and can’t or won’t modify their code. For data stores where a connector already exists, Connect can be used by non-developers who will only need to configure the connectors.

If you need to connect Kafka to a data store and a connector does not exist yet, you can choose between writing an app using the Kafka clients or the Connect APIs. Connect is recommended because it provides out of the box features like configuration management, offset storage, parallelization, error handling, support for different data types and standard management REST API. Writing a small app that connects Kafka to a data store sounds simple, but there are many little details you will need to handle around data types and configuration that make the task non-trivial - Kafka connect handles most of this for you, allowing you to focus on transporting data to and from the external stores.

I realize that is a bit out of scope, but I'd like to recommend Apache NiFi as an alternative to Kafka Connect. See: https://nifi.apache.org/

NiFi is a good alternative to Flume too. NiFi provide an advanced data management capability covering easily Kafka producer or Kafka consumer needs with no coding, visually, at most, some regular expressions.