Support Video: How to run Kafka Mirror Maker using Kerberos clusters?

[#1]

This video provides the steps to configure Kafka Mirror Maker in Kerberized clusters.

Open YouTube video here

To run Kafka Mirror Maker using Kerberos clusters, do the following:

Environment:

Cluster A (Source-Kerberized):

c189-node2.squadron-labs.com Broker1
c189-node3.squadron-labs.com Broker2
c189-node4.squadron-labs.com Broker3

Cluster B (Destination-Kerberized):

c289-node2.squadron-labs.com Broker1
c289-node3.squadron-labs.com Broker2
c289-node4.squadron-labs.com Broker3

* In this example, both environments are sharing same KDC and it was created on specific principal for the same.

Destination Files 

consumer.properties
bootstrap.servers=<brokerSourceHost>:<brokerPort>,<brokerSourceHost>:<brokerPort>
group.id=<consumerGroupName>
security.protocol=PLAINTEXTSASL

producer.properties
bootstrap.servers=<brokerDestinationHost>:<brokerPort>,<brokerDestinationHost>:<brokerPort>
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
security.protocol=PLAINTEXTSASL

kafka_mirrormaker_jaas.conf
KafkaClient {
     com.sun.security.auth.module.Krb5LoginModule required
     useKeyTab=true
     keyTab="<pathTokeytab>"
     storeKey=true
     useTicketCache=false
     serviceName="kafka"
     principal="<principal>@<REALM>";
    };

Destination commands

1. export KAFKA_OPTS="-Djava.security.auth.login.config=<path_to_kafka_mirrormaker_jaas.conf>"

2. ./kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config <path_To_Consumer.properties> 
   --producer.config <path_To_producer.properties> --whitelist "<MirrorMakertopicName>"

Number of consumption streams

Use the --num.streams option to specify the number of mirror consumer threads to create. 

Source commands

1. ./kafka-console-producer.sh --broker-list <brokerSourceHost>:<brokerPort> 
   --topic <MirrorMakertopicName> --security-protocol PLAINTEXTSASL

2. ./kafka-console-consumer.sh --bootstrap-server <brokerDestinationHost>:<brokerPort> 
   --topic <MirrorMakertopicName> --security-protocol PLAINTEXTSASL