Reply
ujj
Explorer
Posts: 7
Registered: ‎05-16-2016

Kafka MirrorMaker

How do I configure KAFKA_MIRRORMAKER via the configuration file? I want to provision my cluster and point it to an existing Kafka cluster so that it can start mirroring data from it.

 

I can't find any documentation around it.

Cloudera Employee
Posts: 67
Registered: ‎02-18-2014

Re: Kafka MirrorMaker

A very late reply, but I was just working on this today. Here's how it's done. Do note, though, that Cloudera Director up through version 2.8.0 does not recognize the KAFKA_MIRROR_MAKER role for validation purposes, but it should still work.

 

1. Include KAFKA in the array of cluster services as usual, and do specify the URL for the Kafka parcel.

 

    products {
      CDH: 5
      KAFKA: 3
    }

    parcelRepositories: ["https://archive.cloudera.com/cdh5/parcels/5.15.0/",
                         "https://archive.cloudera.com/kafka/parcels/3.0/"]

2. Include the role "KAFKA_MIRROR_MAKER" in the desired instance group. I experimented with using the master group.

 

        roles {
            HDFS: [NAMENODE, SECONDARYNAMENODE]
            YARN: [RESOURCEMANAGER, JOBHISTORY]
            KAFKA: [KAFKA_BROKER, KAFKA_MIRROR_MAKER]
        }

3. The mirror maker requires a topic whitelist, a source broker list, and a destination broker list. Those can be set in the Cloudera Director configuration file, but I found that the mirror maker will not start up successfully unless the brokers have resolvable host names (or, I suppose, IP addresses). In all likelihood either the source or destination brokers are in the cluster being created by Director, so unless you are also, say, using bootstrap scripts to establish pre-ordained alternative host names for instances as they come up, the mirror maker won't work immediately. In that case, you can just adjust the configuration properties in Cloudera Manager after Cloudera Director is done to establish the linkage. Cloudera Director will synchronize the changes back to itself.

 

        configs {
            KAFKA {
                KAFKA_MIRROR_MAKER {
                    "source.bootstrap.servers": "myclusterbroker1:9092"
                    "bootstrap.servers": "remotebroker1:9092"
                    whitelist: mytopic1
                }
            }
        }
Announcements