Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

kafka connect

How to install kafka connect in a distributed mnode in HDP. I am using HDP 2.6

4 REPLIES 4

Explorer

@ARUN Are you able to setup kafka-Connect with HDP 2.6. If so, can you please brief the steps.

Super Collaborator

If you are running Kafka 0.10 or newer, connect-distributed.sh exists somewhere under /usr/hdp/current/kafka already.

You can run that process on multiple machines to create a Kafka Connect cluster.

New Contributor

What command did you use to run this and what is the location of connect-distributed.properties location in hortonworks?

Thanks,

Anand

Explorer

Kafka Connect Setup:

  1. Download the Confluent-Kafka tar for Confluent:

https://www.confluent.io/download/
2. Untar the package and copy the '/share' folder under '/usr/hdp/hdp_version_/kafka/' folder
3. update the CLASSPPATH with jars files location, in my case its '/usr/hdp/2.6.4.0-91/kafka/share/java'
4. Make appropriate changes to 'connect-distributed' & 'connect-standalone' property files under /etc/kafka/hdp_version/0/
5. I added 'quickstart-hdfs.properties' under '/etc/kafka/hdp_version/0/' which includes topic names,topics dirs,flush size etc.
6. Run a test job with these changes and worked for me.

**Attaching a template of quickstart hdfs properties file.quickstart-hdfs.txt