Created 08-25-2016 11:50 AM
Hi,
I am following steps on Kafka-Mirror given at "http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_kafka-user-guide/bk_kafka-user-guide-20160628.pdf" and having two separate cluster but when i run kafka.tools.MirrorMaker i am getting below Error:-
[2016-08-25 17:20:00,081] WARN The configuration serializer.class = kafka.serializer.DefaultEncoder was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2016-08-25 17:20:00,136] ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$) org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value. at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:148) at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:49) at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:56) at org.apache.kafka.clients.consumer.ConsumerConfig.<init>(ConsumerConfig.java:336) at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:541) at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:523) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:330) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:328) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Range.foreach(Range.scala:141) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.tools.MirrorMaker$.createNewConsumers(MirrorMaker.scala:328) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:246) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) Exception in thread "main" java.lang.NullPointerException at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276) at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
consumer_mirr.properties
zookeeper.connect=sourceHOST:2181 zookeeper.connection.timeout.ms=6000 group.id=test-consumer-group-mirror consumer.timeout.ms=5000 shallow.iterator.enable=true mirror.topics.whitelist=app_log
producer_mirr.properties
metadata.broker.list=targetHOST:6667 request.required.acks=0 producer.type=async compression.codec=none serializer.class=kafka.serializer.DefaultEncoder queue.enqueue.timeout.ms=-1 max.message.size=1000000 queue.time=1000
Your help is really appriciated here. Thanks in Advance!!!
Created 08-25-2016 11:52 AM
Mirror command below:-
./kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config /usr/hdp/current/kafka-broker/config/consumer_mirr.properties --producer.config /usr/hdp/current/kafka-broker/config/producer_mirr.properties --whitelist MukeshTest --new.consumer
Created 08-25-2016 04:12 PM
Are you using the new Kafka consumer? (The bootstrap.servers property is required for the new consumer.)
Created 08-26-2016 06:43 AM
Thanks @lgeorge for your response.
i have tried with and without new consumer and same error message.
Created 08-25-2016 04:04 PM
There is a bootstrap.servers property (see https://community.hortonworks.com/articles/24599/kafka-mirrormaker.html) but I'm not sure when it's required. I'll see if I can find someone who has more info.