How to setup kafka cluster on hortonworks platform

[#1]

Can some one point me in right direction of how to setup KAFKA cluster on HDP platform.

To setup kafka clusters usually we add broker.id=1,2,3... but hortonworks doesn't provide this option to add in thr..

Issue: is when i try to create kafka topics i am able to create with 2,3 partitions and replicas 2,3.. but starting consumer and producer start throwing exception unknown topic or partition. My guess is it;s due to kafka HA configuration..

[#2 · Accepted solution]

Hey @Pankaj Singh!
Not sure if I get it right, but, how are you consuming/producing to/from kafka topics?
Through kafka-console-consumer/kafka-console-producer?

Could you share with us the output from the following commands?

[root@node2 ~]# kafka-topics.sh --zookeeper node1:2181,node2:2181,node3:2181 --describe --topic vini
Topic:vini  PartitionCount:3  ReplicationFactor:3 Configs:message.timestamp.type=LogAppendTime
  Topic: vini Partition: 0  Leader: 1001  Replicas: 1001,1002,1003  Isr: 1002,1003,1001
  Topic: vini Partition: 1  Leader: 1002  Replicas: 1002,1003,1001  Isr: 1002,1003,1001
  Topic: vini Partition: 2  Leader: 1003  Replicas: 1003,1001,1002  Isr: 1002,1003,1001
[root@node2 ~]# kafka-console-producer.sh --broker-list node2:6667 --topic vini 
>testing testing
>this is a test        
>hcc 
[root@node2 ~]# kafka-console-consumer.sh --bootstrap-server node2:6667 --topic vini --from-beginning
this is a test
testing testing
hcc 
[root@node2 ~]# zookeeper-client
[zk: localhost:2181(CONNECTED) 9] ls /brokers/topics/vini/partitions 
[0, 1, 2]

Hope this helps!

[#3]

I apology for not putting the answer earlier. I was able to resolve the issue. Issue was due to ranger policy setup on Kafka topics.

After turning off the ranger policy it works out of the box. Kafka cluster creates a meta.properties folder. HDP infra setups the Kafka in cluster from start