Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

When there are a lot message the mirror maker is not able to consume from the last saved offset

When there are a lot message the mirror maker is not able to consume from the last saved offset

Explorer

I'm having issue with Kafka Mirror Maker. I've stopped the mirror maker for 30 minutes due to a cluster upgrade and at the restart of the cluster the mirror maker is not able to consume data from the source cluster. I see that the lag of the consumer group of the mirror maker is very high so I'm thinking about some parameters to change in order to increase the buffer size of the mirror maker. I've tried changing the consumer group for the mirror maker and in this case this operation allows to restart consuming data from the latest messages. When I try to restart the process from the last saved offsets I see a peak of consumed data but the mirror maker is not able to commit offsets infact the log is blocked at the row: INFO kafka.tools.MirrorMaker$: Committing offsets and no more rows are showev after this one.

I think that the problem is related to the huge amount of data to process. Ive running a cluster with Kafka 0.8.2.1 with this configuration:

 

auto.offset.reset=largest

offsets.storage=zookeeper

dual.commit.enabled=false

 

2 REPLIES 2

Re: When there are a lot message the mirror maker is not able to consume from the last saved offset

Super Collaborator
Take a look at the following settings:

num.streams
num.producers

Increasing the num.streams will increase the number of consumer threads that you have running and increasing num.producers will allow you to produce more messages to the destination in parallel.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330


-pd

Re: When there are a lot message the mirror maker is not able to consume from the last saved offset

Explorer

Hi, thank you for the responce.

 

Consumer stream is bounded to the number of the partition of the topic so increasing the number of consumer and producer will not solve the problem. From one partition you can have maximum one consumer for a single consumer group.

 

I was thinking about to increase the queue size of the mirror maker but this is still not working..

Don't have an account?
Coming from Hortonworks? Activate your account here