Are Kafka Topics created under zookeeper when cluster is kerberized and Ranger is enabled for authorization?

[#1]

What is the role of zookeeper here with Kafka apart from distributed co-ordination?

Are Kafka Topics created under zookeeper when cluster is kerberized and Ranger is enabled for authorization?

I know the messages are stored under logs.dir path. Does zookeeper store metadata of topics etc? Does zookeeper store topics only when native ACL is used and not in Ranger?

[#2 · Accepted solution]

@Tech Gig

No, they are not created under Zookeeper but Zookeeper is used by Kafka for state management, kafka topics and partitions.

For example, consumers mark the offset for the record they have read to know what record they will be reading next. I'd recommend following deck (slide 10) to see how this works:

https://www.slideshare.net/rahuldausa/apache-kafka-16727853?next_slideshow=1

and slide 15 of following deck to understand how Zookeeper is used by Kafka:

https://www.slideshare.net/rahuldausa/introduction-to-kafka-and-zookeeper

[#3]

hey @mqureshi thanks for providing the information. Apart from other zookeeper uses, so I assume the communication flow is to be like Producer will contact the zookeeper to get broker address and start producing messages to the brokers. Consumer will poll the brokers for messages and update the consumed message offset to Zookeeper.

[#4]

@Tech Gig

Yes. If you see the directory structure in Zookeeper as described below, you will have a more clear idea. But your understanding is right:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper