Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka MirrorMaker - Mirroring different Kafka versions

Kafka MirrorMaker - Mirroring different Kafka versions

Contributor

Hi,

We need to use MirrorMaker to replicate data from 2 Kafka clusters running different versions.

Source ------------------------------> Target

Kafka 0.10.1 -> MirrorMaker -> Kafka 0.9.0

We are able to get this to work running MirrorMaker 0.9.0 using the "old zookeeper" consumer type, but would like to know if this is possible with MirrorMaker 0.10.1? We would like to limit the exposure of the source ZK cluster, hence the need to use the new MirrorMaker 0.10.1 consumer. Are there any consumer/producer configs we can use in MirrorMaker to allow us to use version 0.10.1?
# Exception in MirrorMaker when running version 0.10.1
[2017-08-31 16:43:15,799] ERROR Uncaught error in kafka producer I/O thread:  (org.apache.kafka.clients.producer.internals.Sender) org.apache.kafka.common.protocol.types.SchemaException:
Error reading field 'brokers': Error reading field 'host': Error reading string
of length 26992, only 2176 bytes
Thanks,
Jon
1 REPLY 1
Highlighted

Re: Kafka MirrorMaker - Mirroring different Kafka versions

Contributor

We either need to figure out how to get MirrorMaker 0.9.0 to use the new client api

OR get MirrorMaker 0.10.1 to use the 0.9.0 compatible message format.

Don't have an account?
Coming from Hortonworks? Activate your account here