Support Questions

Find answers, ask questions, and share your expertise

Kafka Mirror Error:- Missing required configuration "bootstrap.servers"

avatar

Hi,

I am following steps on Kafka-Mirror given at "http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_kafka-user-guide/bk_kafka-user-guide-20160628.pdf" and having two separate cluster but when i run kafka.tools.MirrorMaker i am getting below Error:-

[2016-08-25 17:20:00,081] WARN The configuration serializer.class = kafka.serializer.DefaultEncoder was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2016-08-25 17:20:00,136] ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$)
org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value.
        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:148)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:49)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:56)
        at org.apache.kafka.clients.consumer.ConsumerConfig.<init>(ConsumerConfig.java:336)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:541)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:523)
        at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:330)
        at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:328)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
        at scala.collection.immutable.Range.foreach(Range.scala:141)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
        at scala.collection.AbstractTraversable.map(Traversable.scala:105)
        at kafka.tools.MirrorMaker$.createNewConsumers(MirrorMaker.scala:328)
        at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:246)
        at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
Exception in thread "main" java.lang.NullPointerException
        at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276)
        at kafka.tools.MirrorMaker.main(MirrorMaker.scala)


consumer_mirr.properties

zookeeper.connect=sourceHOST:2181
zookeeper.connection.timeout.ms=6000
group.id=test-consumer-group-mirror
consumer.timeout.ms=5000
shallow.iterator.enable=true
mirror.topics.whitelist=app_log

producer_mirr.properties

metadata.broker.list=targetHOST:6667
request.required.acks=0
producer.type=async
compression.codec=none
serializer.class=kafka.serializer.DefaultEncoder
queue.enqueue.timeout.ms=-1
max.message.size=1000000
queue.time=1000

Your help is really appriciated here. Thanks in Advance!!!

4 REPLIES 4

avatar

Mirror command below:-

./kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config /usr/hdp/current/kafka-broker/config/consumer_mirr.properties --producer.config /usr/hdp/current/kafka-broker/config/producer_mirr.properties --whitelist MukeshTest --new.consumer

avatar
Super Collaborator

Are you using the new Kafka consumer? (The bootstrap.servers property is required for the new consumer.)

avatar

Thanks @lgeorge for your response.

i have tried with and without new consumer and same error message.

avatar
Super Collaborator

There is a bootstrap.servers property (see https://community.hortonworks.com/articles/24599/kafka-mirrormaker.html) but I'm not sure when it's required. I'll see if I can find someone who has more info.