Kafka MirrorMaker

[#1]

Introduction

MirrorMaker can be used to replicate messages of a defined list of topics across clusters.

Configuration

We need to configure both the producer and the consumer through config files. The config file for the producer will define the destination cluster Kafka brokers, whereas the consumer one will point to the Zookeeper servers from the source cluster. The following configuration also assumes that security is enabled (Kerberos) so the PLAINTEXTSASL protocol will be used:

mm_producer-1.properties:

bootstrap.servers=dest-kafkabrk1:6667,dest-kafkabrk2:6667,dest-kafkabrk3:6667

producer.type=async

queue.time=1000000

queue.enqueueTimeout.ms=-1

security.protocol=PLAINTEXTSASL

#sasl.kerberos.service.name=kafka

mm_consumer-1.properties:

zookeeper.connect=src-zkhost1:2181,src-zkhost2:2181,src-zkhost3:2181

zk.connectiontimeout.ms=1000000

bootstrap.servers=src-kafkabrk1:6667,src-kafkabrk2:6667,src-kafkabrk3:6667

consumer.timeout.ms=-1

security.protocol=PLAINTEXTSASL

group.id=kafka-mirror

#sasl.kerberos.service.name=kafka

Running MirrorMaker

MirrorMaker will take either whitelist or blacklist, to either define which topics needs mirroring or which ones don't, but only one of these options can be specified. They can be comma-separated or can even use wildcards (*) for the topic names.

For example:

bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config mm_consumer-1.properties --producer.config mm_producer-1.properties --whitelist mm-topic-1,mm-topic-2

[#2]

For additional information, see recent additions to the Kafka Guide. Here's the link for HDP 2.4.2:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_kafka-user-guide/content/ch_kafka_mirrorm...