Starting from version 0.8.2.0, the offsets committed by the consumers aren’t saved in Zookeeper but on a partitioned and replicated topic named __consumer_offsets (internal topic), which is hosted on the Kafka brokers in the cluster. When a consumer commits some offsets, it sends a message to the broker to the __consumer_offsets topic. The message has the following structure : key = [group, topic, partition] value = offset If consumer process dies, it will be able to start up and start reading where it left off based on offset stored in "__consumer_offset" or as discussed another consumer in the consumer group can take over.
... View more
