Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

kafka connect

How to install kafka connect in a distributed mnode in HDP. I am using HDP 2.6

3 REPLIES 3

Explorer

@ARUN Are you able to setup kafka-Connect with HDP 2.6. If so, can you please brief the steps.

Super Collaborator

If you are running Kafka 0.10 or newer, connect-distributed.sh exists somewhere under /usr/hdp/current/kafka already.

You can run that process on multiple machines to create a Kafka Connect cluster.

Explorer

Kafka Connect Setup:

  1. Download the Confluent-Kafka tar for Confluent:

https://www.confluent.io/download/
2. Untar the package and copy the '/share' folder under '/usr/hdp/hdp_version_/kafka/' folder
3. update the CLASSPPATH with jars files location, in my case its '/usr/hdp/2.6.4.0-91/kafka/share/java'
4. Make appropriate changes to 'connect-distributed' & 'connect-standalone' property files under /etc/kafka/hdp_version/0/
5. I added 'quickstart-hdfs.properties' under '/etc/kafka/hdp_version/0/' which includes topic names,topics dirs,flush size etc.
6. Run a test job with these changes and worked for me.

**Attaching a template of quickstart hdfs properties file.quickstart-hdfs.txt