Support Questions

Find answers, ask questions, and share your expertise

Are Kafka Topics created under zookeeper when cluster is kerberized and Ranger is enabled for authorization?

Explorer

What is the role of zookeeper here with Kafka apart from distributed co-ordination?

Are Kafka Topics created under zookeeper when cluster is kerberized and Ranger is enabled for authorization?

I know the messages are stored under logs.dir path. Does zookeeper store metadata of topics etc? Does zookeeper store topics only when native ACL is used and not in Ranger?

1 ACCEPTED SOLUTION

Super Guru
@Tech Gig

No, they are not created under Zookeeper but Zookeeper is used by Kafka for state management, kafka topics and partitions.

For example, consumers mark the offset for the record they have read to know what record they will be reading next. I'd recommend following deck (slide 10) to see how this works:

https://www.slideshare.net/rahuldausa/apache-kafka-16727853?next_slideshow=1

and slide 15 of following deck to understand how Zookeeper is used by Kafka:

https://www.slideshare.net/rahuldausa/introduction-to-kafka-and-zookeeper

View solution in original post

3 REPLIES 3

Super Guru
@Tech Gig

No, they are not created under Zookeeper but Zookeeper is used by Kafka for state management, kafka topics and partitions.

For example, consumers mark the offset for the record they have read to know what record they will be reading next. I'd recommend following deck (slide 10) to see how this works:

https://www.slideshare.net/rahuldausa/apache-kafka-16727853?next_slideshow=1

and slide 15 of following deck to understand how Zookeeper is used by Kafka:

https://www.slideshare.net/rahuldausa/introduction-to-kafka-and-zookeeper

Explorer

hey @mqureshi thanks for providing the information. Apart from other zookeeper uses, so I assume the communication flow is to be like Producer will contact the zookeeper to get broker address and start producing messages to the brokers. Consumer will poll the brokers for messages and update the consumed message offset to Zookeeper.

Super Guru

@Tech Gig

Yes. If you see the directory structure in Zookeeper as described below, you will have a more clear idea. But your understanding is right:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.