Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka Active Active Cluster Setup

Kafka Active Active Cluster Setup

Hi All, 

 

I am trying to set up a ambari cluster for Kafka in setting up as Active-Active cluster. 

What was the best way of setting up this?

Test case: 

Storm application writes to Kafka topic to two clusters ( Cluster A & cluster B which are active-active) both cluster will have the same data and same kafak topic name. when we are consuming this data, from both clusters from a Kafka topic,  how the offset will behave here? 

 

What is the best scenario in setting up this Rededent cluster with active, active, so if one Kafka cluster is down other will be available in reading the data?

 

Any help/workarounds would be appreciated.

 

Thank You, 

Sandeep

 

3 REPLIES 3
Highlighted

Re: Kafka Active Active Cluster Setup

Contributor

HI @sandeep_hadoopa 

 

I believe you can use Kafka mirror maker for this scenario, where you can write data to one cluster and replicate the messages to the other. 

 

https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/kafka-mirroring-data/content/mirroring_data_be...

 

I hope that helps.

Regards,

Manuel.

Highlighted

Re: Kafka Active Active Cluster Setup

Hi @ManuelCalvo

 

Thank your inputs. 

 

I am still not clear about a few things. 

 

Mirror maker in just taking a copy of data from one cluster to another cluster. 

 

but here a scenario. 

My application is writing the data into a Cluster A and on cluster B we set up a mirror maker in copying data from Cluster A to Cluster B. What if Cluster A is down? 

and on the other side,  consuming data we need to either to read from both clusters if cluster A down then we need to consume the data from Cluster B.

Not sure what mechanism in writing and reading the data? 

May be Storm application need to write data to two clusters A, B, and reading need to be the same as well from both cluster. But was not sure how offsets will behave here when consuming the data from both clusters? 

maybe adding an advertise listener with a common IP address can be readable;le from cluster? 

 

 

 

 

Highlighted

Re: Kafka Active Active Cluster Setup

Contributor

@sandeep_hadoopa 

 

I would say that the logic has to be in the application. If this is because of fault-tolerant or high availability, you can use 1 single cluster with multiple hosts and replication factor >= 3 to replicate the topic's data among hosts. If one broker is down, then you have other 2 machines to replace the topic leader and continue working.

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here