question Re: Are Kafka Topics created under zookeeper when cluster is kerberized and Ranger is enabled for authorization? in Archives of Support Questions (Read Only)

Are Kafka Topics created under zookeeper when cluster is kerberized and Ranger is enabled for authorization?

techongig — Sun, 30 Apr 2017 16:29:24 GMT

What is the role of zookeeper here with Kafka apart from distributed co-ordination?

Are Kafka Topics created under zookeeper when cluster is kerberized and Ranger is enabled for authorization?

I know the messages are stored under logs.dir path. Does zookeeper store metadata of topics etc? Does zookeeper store topics only when native ACL is used and not in Ranger?

Re: Are Kafka Topics created under zookeeper when cluster is kerberized and Ranger is enabled for authorization?

mqureshi — Tue, 02 May 2017 01:44:34 GMT

@Tech Gig

No, they are not created under Zookeeper but Zookeeper is used by Kafka for state management, kafka topics and partitions.

For example, consumers mark the offset for the record they have read to know what record they will be reading next. I'd recommend following deck (slide 10) to see how this works:

https://www.slideshare.net/rahuldausa/apache-kafka-16727853?next_slideshow=1

and slide 15 of following deck to understand how Zookeeper is used by Kafka:

https://www.slideshare.net/rahuldausa/introduction-to-kafka-and-zookeeper

Re: Are Kafka Topics created under zookeeper when cluster is kerberized and Ranger is enabled for authorization?

techongig — Tue, 02 May 2017 18:19:08 GMT

hey @mqureshi thanks for providing the information. Apart from other zookeeper uses, so I assume the communication flow is to be like Producer will contact the zookeeper to get broker address and start producing messages to the brokers. Consumer will poll the brokers for messages and update the consumed message offset to Zookeeper.

Re: Are Kafka Topics created under zookeeper when cluster is kerberized and Ranger is enabled for authorization?

mqureshi — Tue, 02 May 2017 20:50:47 GMT

@Tech Gig

Yes. If you see the directory structure in Zookeeper as described below, you will have a more clear idea. But your understanding is right:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper