Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka MirrorMaker - not replicating messages after being brought up

Highlighted

Kafka MirrorMaker - not replicating messages after being brought up

Expert Contributor

Hi All,

I've 2 Kafka clusters (Kafka 10) & I'm trying to test the MirrorMaker functionality.

Here is what i did :

1) I have identical topics Topic1 on 2 Kafka clusters - Cluster1 & Cluster2

2) On Cluster1, I publish 100 messages on Topic1

3) I've 2 consumers reading messages from the 2 topics on Cluster1 & Cluster2

4) I start MirrorMaker AFTER THE MESSAGES HAVE BEEN PUBLISHED ON Cluster1

$KAFKA10_HOME1/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config $KAFKA10_HOME1/config/mmConsumer.config --num.streams 3 --producer.config $KAFKA10_HOME1/config/mmProducer.config --whitelist="mmtopic1" --abort.on.send.failure true 

I expected that Once the MirrorMaker was started, the 100 messages published on Cluster1, would be published on Clueter2

However that is not happening...

What needs to be done to enable this ?

Pls see my consumer & producer configs.

---------------------------------------------------------------

mmConsumer.config :

bootstrap.servers=localhost:6092,localhost:6093,localhost:6094,localhost:6095

group.id=mmgroup

exclude.internal.topics=true

#mirror.topics.whitelist=mmtopic1

client.id=mmconsumer

auto.commit.enabled=false

mmProducer.config :

bootstrap.servers=localhost:5092,localhost:5093,localhost:5094,localhost:5095

acks=1

batch.size=100

client.id=mmproducer

max.in.flight.requests.per.connection=1

#retries=Int.MaxValue

retries=1000000

block.on.buffer.full=true

1 REPLY 1

Re: Kafka MirrorMaker - not replicating messages after being brought up

Expert Contributor

@mqureshi, @Sriharsha Chintalapani , @Artem Ervits -- any ideas on this ?

Don't have an account?
Coming from Hortonworks? Activate your account here