question Re: kafka logs file - what type of details this topic __consumer_offsets holds ? in Archives of Support Questions (Read Only)

kafka logs file - what type of details this topic __consumer_offsets holds ?

amit_dass — Tue, 07 Nov 2017 23:15:33 GMT

Do we know for what purpose these files which will be created after starting Kafka server and all contains __consumer_offsets information in log .

files are : state-change.log ,kafka-request.log,kafka-authorizer.log,controller.log,server.log,log-cleaner.log

I have read that "__consumer_offsets" is the topic what type of details it holds ?

Re: kafka logs file - what type of details this topic __consumer_offsets holds ?

NikhilSilsarma — Mon, 13 Nov 2017 23:17:29 GMT

Starting from version 0.8.2.0, the offsets committed by the consumers aren’t saved in Zookeeper but on a partitioned and replicated topic named __consumer_offsets (internal topic), which is hosted on the Kafka brokers in the cluster.

When a consumer commits some offsets, it sends a message to the broker to the __consumer_offsets topic. The message has the following structure :

key = [group, topic, partition]

value = offset

If consumer process dies, it will be able to start up and start reading where it left off based on offset stored in "__consumer_offset" or as discussed another consumer in the consumer group can take over.

Re: kafka logs file - what type of details this topic __consumer_offsets holds ?

amit_dass — Mon, 13 Nov 2017 23:35:14 GMT

Do we know default partitions and replications for this topic __consumer_offsets (internal topic) ? By any chance if we loses this topic then do we have anything other then this to recover ?

As you explained above , what I understand is now there is no current offset and commit offset concepts .

Please add your valuable knowledge if it is not ?

Re: kafka logs file - what type of details this topic __consumer_offsets holds ?

NikhilSilsarma — Tue, 14 Nov 2017 01:46:11 GMT

you can set the values for the follwoing attributes.

offsets.topic.num.partitions=50

offsets.topic.replication.factor=3

losing this topic may cause issue for kafka broker to startup of service consumer request(metadata fetch failure)