I am facing a weird problem
where my mirror-maker is able to mirror the topic name in the destination
cluster but I am seeing no messages when I try to consume messages from it.
Below is my configuration file – Kafka version : kafka_10.0.1 All Certs /Keys are my
organization issued certificate so no Error for SSL Source Cluster : ================ Zookeeeper Properties: dataDir=/home/kafka/zookeeper
clientPort=2181
maxClientCnxns=3
Server Properties broker.id=1
listeners= SSL://123.456.789.110:9092
advertised.listeners=SSL://123.456.789.110:9092
security.inter.broker.protocol=SSL
auto.create.topics.enable=true
ssl.client.auth=none
ssl.keystore.location=/etc/keys/keystore
ssl.keystore.password=test1234
ssl.truststore.location=/etc/CApath/trustedstore
ssl.truststore.password=test1234
ssl.enabled.protocols=TLSv1.2
log.dirs=/home/kafka/kafka-logs
zookeeper.connect=123.456.789.110:2181
Kafka Up and Running / Log
persistently logging in Source System Target Cluster Zookeeeper Properties dataDir=/home/kafka/zookeeper
clientPort=2181
maxClientCnxns=3
Server Properties broker.id=1
listeners= SSL:// 234.456.789.110:9092
advertised.listeners=SSL://234.456.789.110:9092
security.inter.broker.protocol=SSL
ssl.client.auth=none
ssl.keystore.location=/etc/keys/keystore
ssl.keystore.password=test1234
ssl.truststore.location=/etc/CApath/trustedstore
ssl.truststore.password=test1234
ssl.enabled.protocols=TLSv1.2
ssl.keystore.type = JKS
ssl.truststore.type = JKS
log.dirs=/home/kafka/kafka-logs
zookeeper.connect=234.456.789.110:2181
zookeeper.connection.timeout.ms=6000
mm_consumer.properties : bootstrap.servers=123.456.789.110:9092
exclude.internal.topics = true
auto.offset.reset=earliest
security.protocol = SSL
ssl.truststore.location=/etc/CApath/trustedstore
ssl.truststore.password=Test1234
ssl.enabled.protocols= TLSv1.2
ssl.keystore.location=/etc/keys/keystore
ssl.keystore.password=Test1234
ssl.truststore.type=JKS
ssl.keystore.type=JKS
zookeeper.connection.timeout.ms=6000
group.id=test-consumer-group
client.id=mirror_maker_consumer
consumer.timeout.ms=50000
mm_producer.properties.properties : bootstrap.servers=234.456.789.110:9092
acks=1
batch.size=100
security.protocol = SSL
ssl.truststore.location=/etc/CApath/trustedstore
ssl.truststore.password=Test1234
ssl.enabled.protocols=TLSv1.2
ssl.keystore.location=/etc/keys/keystore
ssl.keystore.password=Test1234
ssl.truststore.type=JKS
ssl.keystore.type=JKS
compression.type=gzip
I then start my mirror maker
with the following command This command run fines and topic APP-logging created in Target cluster but not logs on target
Cluster. kafka]# bin/kafka-run-class.sh kafka.tools.MirrorMaker --new.consumer --consumer.config config/mm_consumer.properties --producer.config config/mm_producer.properties --whitelist * I executed started another
mirror maker command using topic name with below command , This command fails on target hostCluster /bin/kafka-mirror-maker.sh --new.consumer --consumer.config config/mm_consumer.properties --producer.config config/mm_producer.properties --whitelist APP-logging
[2018-06-22 03:59:27,536] FATAL [mirrormaker-thread-0] Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
java.lang.IllegalArgumentException: Invalid timestamp -1
at org.apache.kafka.clients.producer.ProducerRecord.<init>(ProducerRecord.java:60)
at kafka.tools.MirrorMaker$defaultMirrorMakerMessageHandler$.handle(MirrorMaker.scala:678)
at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:414)
[2018-06-22 03:59:27,563] FATAL [mirrormaker-thread-0] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)
Also run the ".*"
including the quotes instead of just *. bin/kafka-run-class.sh kafka.tools.MirrorMaker --new.consumer --consumer.config config/mm_consumer.properties --producer.config config/mm_producer.properties --whitelist ".*"
[2018-06-22 04:22:01,647] FATAL [mirrormaker-thread-0] Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
java.lang.IllegalArgumentException: Invalid timestamp -1
at org.apache.kafka.clients.producer.ProducerRecord.<init>(ProducerRecord.java:60)
at kafka.tools.MirrorMaker$defaultMirrorMakerMessageHandler$.handle(MirrorMaker.scala:678)
at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:414)
[2018-06-22 04:22:01,674] FATAL [mirrormaker-thread-0] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)
Am I missing something?
... View more
