Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka MirrorMaker

Highlighted

Kafka MirrorMaker

Explorer

How do I configure KAFKA_MIRRORMAKER via the configuration file? I want to provision my cluster and point it to an existing Kafka cluster so that it can start mirroring data from it.

 

I can't find any documentation around it.

1 REPLY 1

Re: Kafka MirrorMaker

Expert Contributor

A very late reply, but I was just working on this today. Here's how it's done. Do note, though, that Cloudera Director up through version 2.8.0 does not recognize the KAFKA_MIRROR_MAKER role for validation purposes, but it should still work.

 

1. Include KAFKA in the array of cluster services as usual, and do specify the URL for the Kafka parcel.

 

    products {
      CDH: 5
      KAFKA: 3
    }

    parcelRepositories: ["https://archive.cloudera.com/cdh5/parcels/5.15.0/",
                         "https://archive.cloudera.com/kafka/parcels/3.0/"]

2. Include the role "KAFKA_MIRROR_MAKER" in the desired instance group. I experimented with using the master group.

 

        roles {
            HDFS: [NAMENODE, SECONDARYNAMENODE]
            YARN: [RESOURCEMANAGER, JOBHISTORY]
            KAFKA: [KAFKA_BROKER, KAFKA_MIRROR_MAKER]
        }

3. The mirror maker requires a topic whitelist, a source broker list, and a destination broker list. Those can be set in the Cloudera Director configuration file, but I found that the mirror maker will not start up successfully unless the brokers have resolvable host names (or, I suppose, IP addresses). In all likelihood either the source or destination brokers are in the cluster being created by Director, so unless you are also, say, using bootstrap scripts to establish pre-ordained alternative host names for instances as they come up, the mirror maker won't work immediately. In that case, you can just adjust the configuration properties in Cloudera Manager after Cloudera Director is done to establish the linkage. Cloudera Director will synchronize the changes back to itself.

 

        configs {
            KAFKA {
                KAFKA_MIRROR_MAKER {
                    "source.bootstrap.servers": "myclusterbroker1:9092"
                    "bootstrap.servers": "remotebroker1:9092"
                    whitelist: mytopic1
                }
            }
        }