Support Questions
Find answers, ask questions, and share your expertise

best practice for kafka replication

New Contributor

my application are working at three region center.

i would like to replicate every kafka topic mutually

center A to center B and C,center B to center A and C,center C to center A and center B.

I am worrying about which to do. use mirror maker or simply publish and consume nifi processor.

Have you decided to decide?

3 REPLIES 3

Hi @sas yuki For this, and many other best practice style questions when it comes to Kafka, may I recommend the recent DataWorks summit session on Kafka Best Practices:

Streamed video: https://www.youtube.com/watch?v=maD_7ZdyuAU

Slides: https://www.slideshare.net/HadoopSummit/apache-kafka-best-practices

Hope that helps!

So each data center is the primary producer for one topic, and you want to end up with topics A, B, C at each data center?

Something like this...

Data Center A

  • Topic A (primary)
  • Topic B (replicated)
  • Topic C (replicated)

Data Center B

  • Topic B (primary)
  • Topic A (replicated)
  • Topic C (replicated)

Data Center C

  • Topic C (primary)
  • Topic A (replicated)
  • Topic B (replicated)

New Contributor

Hi,

I have same issue, but i'm not sure that i understand @Bryan Bende's reply.
If we have 3 DC:

DC 1:

BR 1 ( Leader) 5( T1, T2)
BR 2 ( T2)
BR 3

DC 1:

BR 1
BR 2
BR 3

DC 1:

BR 1
BR 2
BR 3

Question's, how to force T2 ( Topic 2) to replicate on choosen node? or choosen DC and how kafka will now that there's two DC.
Actually, i have ZK and KF configured respectively on different DC ( 6 KF & 7 ZK).

Thanks,