I'm having issue with Kafka Mirror Maker. I've stopped the mirror maker for 30 minutes due to a cluster upgrade and at the restart of the cluster the mirror maker is not able to consume data from the source cluster. I see that the lag of the consumer group of the mirror maker is very high so I'm thinking about some parameters to change in order to increase the buffer size of the mirror maker. I've tried changing the consumer group for the mirror maker and in this case this operation allows to restart consuming data from the latest messages. When I try to restart the process from the last saved offsets I see a peak of consumed data but the mirror maker is not able to commit offsets infact the log is blocked at the row: INFO kafka.tools.MirrorMaker$: Committing offsets and no more rows are showev after this one.
I think that the problem is related to the huge amount of data to process. Ive running a cluster with Kafka 0.8.2.1 with this configuration:
Hi, thank you for the responce.
Consumer stream is bounded to the number of the partition of the topic so increasing the number of consumer and producer will not solve the problem. From one partition you can have maximum one consumer for a single consumer group.
I was thinking about to increase the queue size of the mirror maker but this is still not working..