Support Questions

Find answers, ask questions, and share your expertise

kafka logs file - what type of details this topic __consumer_offsets holds ?

avatar
Expert Contributor

Do we know for what purpose these files which will be created after starting Kafka server and all contains __consumer_offsets information in log .

files are : state-change.log ,kafka-request.log,kafka-authorizer.log,controller.log,server.log,log-cleaner.log

I have read that "__consumer_offsets" is the topic what type of details it holds ?



1 ACCEPTED SOLUTION

avatar
Contributor

Starting from version 0.8.2.0, the offsets committed by the consumers aren’t saved in Zookeeper but on a partitioned and replicated topic named __consumer_offsets (internal topic), which is hosted on the Kafka brokers in the cluster.

When a consumer commits some offsets, it sends a message to the broker to the __consumer_offsets topic. The message has the following structure :

key = [group, topic, partition]

value = offset

If consumer process dies, it will be able to start up and start reading where it left off based on offset stored in "__consumer_offset" or as discussed another consumer in the consumer group can take over.

View solution in original post

3 REPLIES 3

avatar
Contributor

Starting from version 0.8.2.0, the offsets committed by the consumers aren’t saved in Zookeeper but on a partitioned and replicated topic named __consumer_offsets (internal topic), which is hosted on the Kafka brokers in the cluster.

When a consumer commits some offsets, it sends a message to the broker to the __consumer_offsets topic. The message has the following structure :

key = [group, topic, partition]

value = offset

If consumer process dies, it will be able to start up and start reading where it left off based on offset stored in "__consumer_offset" or as discussed another consumer in the consumer group can take over.

avatar
Expert Contributor

Do we know default partitions and replications for this topic __consumer_offsets (internal topic) ? By any chance if we loses this topic then do we have anything other then this to recover ?

As you explained above , what I understand is now there is no current offset and commit offset concepts .

Please add your valuable knowledge if it is not ?



avatar
Contributor

you can set the values for the follwoing attributes.

offsets.topic.num.partitions=50

offsets.topic.replication.factor=3

losing this topic may cause issue for kafka broker to startup of service consumer request(metadata fetch failure)