Kafka (0.10.0.1) Mirror Maker is not working, seems the process is running but is doing nothing

[#1]

I am facing a weird problem where my mirror-maker is able to mirror the topic name in the destination cluster but I am seeing no messages when I try to consume messages from it. Below is my configuration file –

Kafka version : kafka_10.0.1

All Certs /Keys are my organization issued certificate so no Error for SSL

Source Cluster :

================

Zookeeeper Properties:

dataDir=/home/kafka/zookeeper
clientPort=2181
maxClientCnxns=3

Server Properties

broker.id=1
listeners= SSL://123.456.789.110:9092
advertised.listeners=SSL://123.456.789.110:9092
security.inter.broker.protocol=SSL
auto.create.topics.enable=true
ssl.client.auth=none
ssl.keystore.location=/etc/keys/keystore
ssl.keystore.password=test1234
ssl.truststore.location=/etc/CApath/trustedstore
ssl.truststore.password=test1234
ssl.enabled.protocols=TLSv1.2
log.dirs=/home/kafka/kafka-logs
zookeeper.connect=123.456.789.110:2181

Kafka Up and Running / Log persistently logging in Source System

Target Cluster

Zookeeeper Properties

dataDir=/home/kafka/zookeeper
clientPort=2181
maxClientCnxns=3

Server Properties

broker.id=1
listeners= SSL:// 234.456.789.110:9092
advertised.listeners=SSL://234.456.789.110:9092
security.inter.broker.protocol=SSL
ssl.client.auth=none
ssl.keystore.location=/etc/keys/keystore
ssl.keystore.password=test1234
ssl.truststore.location=/etc/CApath/trustedstore
ssl.truststore.password=test1234
ssl.enabled.protocols=TLSv1.2
ssl.keystore.type = JKS
ssl.truststore.type = JKS
log.dirs=/home/kafka/kafka-logs
zookeeper.connect=234.456.789.110:2181
zookeeper.connection.timeout.ms=6000

mm_consumer.properties :

bootstrap.servers=123.456.789.110:9092
exclude.internal.topics = true
auto.offset.reset=earliest
security.protocol = SSL
ssl.truststore.location=/etc/CApath/trustedstore
ssl.truststore.password=Test1234
ssl.enabled.protocols= TLSv1.2
ssl.keystore.location=/etc/keys/keystore
ssl.keystore.password=Test1234
ssl.truststore.type=JKS
ssl.keystore.type=JKS
zookeeper.connection.timeout.ms=6000
group.id=test-consumer-group
client.id=mirror_maker_consumer
consumer.timeout.ms=50000

mm_producer.properties.properties :

bootstrap.servers=234.456.789.110:9092
acks=1
batch.size=100
security.protocol = SSL
ssl.truststore.location=/etc/CApath/trustedstore
ssl.truststore.password=Test1234
ssl.enabled.protocols=TLSv1.2
ssl.keystore.location=/etc/keys/keystore
ssl.keystore.password=Test1234
ssl.truststore.type=JKS
ssl.keystore.type=JKS
compression.type=gzip

I then start my mirror maker with the following command

This command run fines and topic APP-logging created in Target cluster but not logs on target Cluster.

kafka]# bin/kafka-run-class.sh kafka.tools.MirrorMaker --new.consumer --consumer.config config/mm_consumer.properties --producer.config config/mm_producer.properties --whitelist *

I executed started another mirror maker command using topic name with below command , This command fails on target hostCluster

/bin/kafka-mirror-maker.sh --new.consumer --consumer.config config/mm_consumer.properties --producer.config config/mm_producer.properties --whitelist APP-logging

[2018-06-22 03:59:27,536] FATAL [mirrormaker-thread-0] Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
java.lang.IllegalArgumentException: Invalid timestamp -1
at org.apache.kafka.clients.producer.ProducerRecord.<init>(ProducerRecord.java:60)
at kafka.tools.MirrorMaker$defaultMirrorMakerMessageHandler$.handle(MirrorMaker.scala:678)
at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:414)
[2018-06-22 03:59:27,563] FATAL [mirrormaker-thread-0] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

Also run the ".*" including the quotes instead of just *.

bin/kafka-run-class.sh kafka.tools.MirrorMaker --new.consumer --consumer.config config/mm_consumer.properties --producer.config config/mm_producer.properties --whitelist ".*"

[2018-06-22 04:22:01,647] FATAL [mirrormaker-thread-0] Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
java.lang.IllegalArgumentException: Invalid timestamp -1
at org.apache.kafka.clients.producer.ProducerRecord.<init>(ProducerRecord.java:60)
at kafka.tools.MirrorMaker$defaultMirrorMakerMessageHandler$.handle(MirrorMaker.scala:678)
at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:414)
[2018-06-22 04:22:01,674] FATAL [mirrormaker-thread-0] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

Am I missing something?

[#2]

Hi @VISHAL SINGH!
I'm not a specialist in Kafka, but I guess you're hitting the https://issues.apache.org/jira/browse/KAFKA-4073
As you can see on the link below, it's passing an invalid timestamp to the ProducerRecord class https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/P...

If you're able to test a workaround, can try to change the message.timestamp.type from topics config.

Here's mine e.g.:

[root@node3 bin]# ./kafka-configs.sh --zookeeper node1:2181,node2:2181,node3:2181 --entity-type topics --entity-name vini --describe
Configs for topic 'vini' are 
[root@node3 bin]# ./kafka-configs.sh --zookeeper node1:2181,node2:2181,node3:2181 --entity-type topics --entity-name vini --alter --add-config message.timestamp.type=LogAppendTime
Completed Updating config for entity: topic 'vini'.
[root@node3 bin]# ./kafka-configs.sh --zookeeper node1:2181,node2:2181,node3:2181 --entity-type topics --entity-name vini --describe
Configs for topic 'vini' are message.timestamp.type=LogAppendTime<br>

Details on the link below:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message

PS: I'd test this on a non-production topic, so you can rollback the config whenever you want :)

Hope this helps! 🙂

[#3]

Thanks @Vinicius Higa Murakami for your response .

I have taken Kafka version to 0.10.0.2 in my development environment "timestamp -1" issue got resolved . Now after running Mirror Maker Logs are coming to target Cluster.

[#4]

Good to know @VISHAL SINGH!
Please allow me to ask you, if this issue got solved, it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button. That way other HCC users can quickly find the solution when they encounter the same issue. 🙂