kafka logs file - what type of details this topic __consumer_offsets holds ?

[#1]

Do we know for what purpose these files which will be created after starting Kafka server and all contains __consumer_offsets information in log .

files are : state-change.log ,kafka-request.log,kafka-authorizer.log,controller.log,server.log,log-cleaner.log

I have read that "__consumer_offsets" is the topic what type of details it holds ?

[#2 · Accepted solution]

Starting from version 0.8.2.0, the offsets committed by the consumers aren’t saved in Zookeeper but on a partitioned and replicated topic named __consumer_offsets (internal topic), which is hosted on the Kafka brokers in the cluster.

When a consumer commits some offsets, it sends a message to the broker to the __consumer_offsets topic. The message has the following structure :

key = [group, topic, partition]

value = offset

If consumer process dies, it will be able to start up and start reading where it left off based on offset stored in "__consumer_offset" or as discussed another consumer in the consumer group can take over.

[#3]

Do we know default partitions and replications for this topic __consumer_offsets (internal topic) ? By any chance if we loses this topic then do we have anything other then this to recover ?

As you explained above , what I understand is now there is no current offset and commit offset concepts .

Please add your valuable knowledge if it is not ?

[#4]

you can set the values for the follwoing attributes.

offsets.topic.num.partitions=50

offsets.topic.replication.factor=3

losing this topic may cause issue for kafka broker to startup of service consumer request(metadata fetch failure)