How to install kafka connect in a distributed mnode in HDP. I am using HDP 2.6
If you are running Kafka 0.10 or newer, connect-distributed.sh exists somewhere under /usr/hdp/current/kafka already.
You can run that process on multiple machines to create a Kafka Connect cluster.
Kafka Connect Setup:
2. Untar the package and copy the '/share' folder under '/usr/hdp/hdp_version_/kafka/' folder
3. update the CLASSPPATH with jars files location, in my case its '/usr/hdp/18.104.22.168-91/kafka/share/java'
4. Make appropriate changes to 'connect-distributed' & 'connect-standalone' property files under /etc/kafka/hdp_version/0/
5. I added 'quickstart-hdfs.properties' under '/etc/kafka/hdp_version/0/' which includes topic names,topics dirs,flush size etc.
6. Run a test job with these changes and worked for me.
**Attaching a template of quickstart hdfs properties file.quickstart-hdfs.txt