Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

kafka connect

Highlighted

kafka connect

How to install kafka connect in a distributed mnode in HDP. I am using HDP 2.6

3 REPLIES 3
Highlighted

Re: kafka connect

Explorer

@ARUN Are you able to setup kafka-Connect with HDP 2.6. If so, can you please brief the steps.

Highlighted

Re: kafka connect

Super Collaborator

If you are running Kafka 0.10 or newer, connect-distributed.sh exists somewhere under /usr/hdp/current/kafka already.

You can run that process on multiple machines to create a Kafka Connect cluster.

Highlighted

Re: kafka connect

Explorer

Kafka Connect Setup:

  1. Download the Confluent-Kafka tar for Confluent:

https://www.confluent.io/download/
2. Untar the package and copy the '/share' folder under '/usr/hdp/hdp_version_/kafka/' folder
3. update the CLASSPPATH with jars files location, in my case its '/usr/hdp/2.6.4.0-91/kafka/share/java'
4. Make appropriate changes to 'connect-distributed' & 'connect-standalone' property files under /etc/kafka/hdp_version/0/
5. I added 'quickstart-hdfs.properties' under '/etc/kafka/hdp_version/0/' which includes topic names,topics dirs,flush size etc.
6. Run a test job with these changes and worked for me.

**Attaching a template of quickstart hdfs properties file.quickstart-hdfs.txt

Don't have an account?
Coming from Hortonworks? Activate your account here