Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka Mirror Maker - Errors/Warnings

Kafka Mirror Maker - Errors/Warnings

Expert Contributor

Hi All - i've setup Kafka MirrorMaker using link - Kafka Mirror Maker best Practices,

and am getting the following warnings ->

[2017-06-20 12:29:31,381] WARN The configuration queue.time = 1000000 was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2017-06-20 12:29:31,381] WARN The configuration producer.type = async was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2017-06-20 12:29:31,381] WARN The configuration queue.enqueueTimeout.ms = -1 was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [2017-06-20 12:29:31,460] WARN Property bootstrap.servers is not valid (kafka.utils.VerifiableProperties) [2017-06-20 12:29:31,460] WARN Property zk.connectiontimeout.ms is not valid (kafka.utils.VerifiableProperties) [2017-06-20 12:29:31,627] WARN Property bootstrap.servers is not valid (kafka.utils.VerifiableProperties) [2017-06-20 12:29:31,627] WARN Property zk.connectiontimeout.ms is not valid (kafka.utils.VerifiableProperties) [2017-06-20 12:29:31,635] WARN Property bootstrap.servers is not valid (kafka.utils.VerifiableProperties) [2017-06-20 12:29:31,636] WARN Property zk.connectiontimeout.ms is not valid (kafka.utils.VerifiableProperties)

Kafka version - 0.9

Producer Config ->

bootstrap.servers=localhost:7092,localhost:7093,localhost:7094,localhost:7095

producer.type=async

queue.time=1000000

queue.enqueueTimeout.ms=-1

Consumer Config :

zookeeper.connect=localhost:2181

zk.connectiontimeout.ms=1000000

bootstrap.servers=localhost:9092,localhost:9093,localhost:9094,localhost:9095

consumer.timeout.ms=-1

#security.protocol=PLAINTEXTSASL

group.id=kafka-mirror

Command used to start Mirror Maker :

$KAFKA_HOME1/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config $KAFKA_HOME1/config/mmConsumer.config --num.streams 3 --producer.config $KAFKA_HOME1/config/mmProducer.config --whitelist="mmtopic" --abort.on.send.failure true

Any ideas what needs to be done to remove these warning .. i'm trying to test the optimizations specified in the link above.

4 REPLIES 4

Re: Kafka Mirror Maker - Errors/Warnings

Expert Contributor

@Sriharsha Chintalapani , @mqureshi - any ideas on this.. this is using the link published by Sriharsha - kafka-mirror-maker-best-practices

Highlighted

Re: Kafka Mirror Maker - Errors/Warnings

Expert Contributor

@Sriharsha Chintalapani, @mqureshi -

Hi All - here is the update on this.I was able to fix the following warnings -
1) WARN Property bootstrap.servers is not valid (kafka.utils.VerifiableProperties)

-> removed bootstrap.servers from mmConsumer.config (IT IS REQUIRED ONLY IN mmProducer.config)

2) zk.connectiontimeout.ms is not valid (kafka.utils.VerifiableProperties)   
replaced zk.connectiontimeout.ms with -> zookeeper.connection.timeout.ms=1000000 (in mmConsumer.config)
3) #queue.enqueueTimeout.ms=-1

replaced with queue.enqueue.timeout.ms=-1 (in mmProducer.config)

However, couple are still pending ->

a) producer.type = async was supplied but isn't a known config

How do i ensure the producer is an "async" producer ?

b) queue.time = 1000000 was supplied but isn't a known config

What needs to be done for this ?

Re: Kafka Mirror Maker - Errors/Warnings

@Karan Alang

Are you using kafka 0.9 and are you trying to use new producers in mirror maker?

Best practices written in mind with new client APIs. Configs are applied to Kafka 0.10.x onwards.

Re: Kafka Mirror Maker - Errors/Warnings

Expert Contributor

@Sriharsha Chintalapani -

thanks for the reply..

I'm using Kafka 0.9 & the using the following command to start MirrorMarker ->

$KAFKA_HOME1/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config $KAFKA_HOME1/config/mmConsumer.config --num.streams 3 --producer.config $KAFKA_HOME1/config/mmProducer.config --whitelist="mmtopic" --abort.on.send.failure true

Does the above command use the old Kafka consumer or the new one?

How do i ensure it uses new Kafka consumer/producer & therefore uses the configs supplied in best practices published ?

Don't have an account?
Coming from Hortonworks? Activate your account here