kafka connect

[#1]

How to install kafka connect in a distributed mnode in HDP. I am using HDP 2.6

[#2]

@ARUN Are you able to setup kafka-Connect with HDP 2.6. If so, can you please brief the steps.

[#3]

If you are running Kafka 0.10 or newer, connect-distributed.sh exists somewhere under /usr/hdp/current/kafka already.

You can run that process on multiple machines to create a Kafka Connect cluster.

[#4]

What command did you use to run this and what is the location of connect-distributed.properties location in hortonworks?

Thanks,

Anand

[#5]

Kafka Connect Setup:

1. Download the Confluent-Kafka tar for Confluent:

https://www.confluent.io/download/
2. Untar the package and copy the '/share' folder under '/usr/hdp/hdp_version_/kafka/' folder
3. update the CLASSPPATH with jars files location, in my case its '/usr/hdp/2.6.4.0-91/kafka/share/java'
4. Make appropriate changes to 'connect-distributed' & 'connect-standalone' property files under /etc/kafka/hdp_version/0/
5. I added 'quickstart-hdfs.properties' under '/etc/kafka/hdp_version/0/' which includes topic names,topics dirs,flush size etc.
6. Run a test job with these changes and worked for me.

**Attaching a template of quickstart hdfs properties file.quickstart-hdfs.txt