Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Explorer

Introduction

MirrorMaker can be used to replicate messages of a defined list of topics across clusters.

Configuration

We need to configure both the producer and the consumer through config files. The config file for the producer will define the destination cluster Kafka brokers, whereas the consumer one will point to the Zookeeper servers from the source cluster. The following configuration also assumes that security is enabled (Kerberos) so the PLAINTEXTSASL protocol will be used:

mm_producer-1.properties:

bootstrap.servers=dest-kafkabrk1:6667,dest-kafkabrk2:6667,dest-kafkabrk3:6667

producer.type=async

queue.time=1000000

queue.enqueueTimeout.ms=-1

security.protocol=PLAINTEXTSASL

#sasl.kerberos.service.name=kafka

mm_consumer-1.properties:

zookeeper.connect=src-zkhost1:2181,src-zkhost2:2181,src-zkhost3:2181

zk.connectiontimeout.ms=1000000

bootstrap.servers=src-kafkabrk1:6667,src-kafkabrk2:6667,src-kafkabrk3:6667

consumer.timeout.ms=-1

security.protocol=PLAINTEXTSASL

group.id=kafka-mirror

#sasl.kerberos.service.name=kafka

Running MirrorMaker

MirrorMaker will take either whitelist or blacklist, to either define which topics needs mirroring or which ones don't, but only one of these options can be specified. They can be comma-separated or can even use wildcards (*) for the topic names.

For example:

bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config mm_consumer-1.properties --producer.config mm_producer-1.properties --whitelist mm-topic-1,mm-topic-2

7,253 Views
Comments
Expert Contributor

For additional information, see recent additions to the Kafka Guide. Here's the link for HDP 2.4.2:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_kafka-user-guide/content/ch_kafka_mirrorm...

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎03-26-2016 12:58 PM
Updated by:
 
Contributors
Top Kudoed Authors