Support Questions

Find answers, ask questions, and share your expertise

kafka connect

How to install kafka connect in a distributed mnode in HDP. I am using HDP 2.6



@ARUN Are you able to setup kafka-Connect with HDP 2.6. If so, can you please brief the steps.

Super Collaborator

If you are running Kafka 0.10 or newer, exists somewhere under /usr/hdp/current/kafka already.

You can run that process on multiple machines to create a Kafka Connect cluster.

New Contributor

What command did you use to run this and what is the location of location in hortonworks?




Kafka Connect Setup:

  1. Download the Confluent-Kafka tar for Confluent:
2. Untar the package and copy the '/share' folder under '/usr/hdp/hdp_version_/kafka/' folder
3. update the CLASSPPATH with jars files location, in my case its '/usr/hdp/'
4. Make appropriate changes to 'connect-distributed' & 'connect-standalone' property files under /etc/kafka/hdp_version/0/
5. I added '' under '/etc/kafka/hdp_version/0/' which includes topic names,topics dirs,flush size etc.
6. Run a test job with these changes and worked for me.

**Attaching a template of quickstart hdfs properties file.quickstart-hdfs.txt