Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Re: Zookeeper Setup for 2 Rack Data center

Re: Zookeeper Setup for 2 Rack Data center

Champion
Zookeeper does require an odd number and to continue to work needs a quorum. Because of this any even number of racks will always be in the situation of only supporting the failure of one of the two racks.

Case A: 7 ZK nodes, 4 in rack1 and 3 in rack2. Rack2 can fail but rack1 cannot.

CaseB: 11 ZK nodes, 5 in rack1 and 6 in rack2. Rack1 can fail but rack2 cannot.

Case C: 5 ZK nodes, 1 in rack1, 2 in rack2, and 2 in rack3. You can withstand a single rack failure for any rack. You cannot withstand two rack failures.

The parred down version is minimum 3 racks and 3 ZK nodes. Scaling up beyond this would be based on your need for more ZK znodes and transactions.
7 REPLIES 7

Re: Zookeeper Setup for 2 Rack Data center

Explorer

Hi,

 

Thanks for your response. So can i say that Kafka which is managed zookeeper. is not a right messaging platform for 2 Rack network infrastructure to achieve seamless rack fault tolerant mechanism. Or any other mechanism exists ?

 

Re: Zookeeper Setup for 2 Rack Data center

Champion
I would say that determining your message queuing system based on only have 2 racks, ever, would not be very strategic.

Either accept the failure domain of a singular, specific rack, or use a minimum of 3 racks.

If the concern is that you will have a small number of Kafka brokers, not enough to fill up three racks or the costs of three racks versus two, consider running ZK in a virtualized environment.

Re: Zookeeper Setup for 2 Rack Data center

Explorer

Hi,

 

Thanks again. This gives me an idea of setting up Kafka cluster in VM environment which is then connected to my actual existing servers.

How about Kafka mirroring ? Can these be useful concept of achieving fault tolerance in case of rack failure ? And how will zookeeper ensemble looks in it ? Will it be standalone on both the rack and the target rack zookeeper gets activated when source rack goes down ?

Re: Zookeeper Setup for 2 Rack Data center

Super Collaborator

Not sure I understand correctly your question about zookeeper.

 

If you want to be able to survive a rack failure then you need to spread your zookeeper cluster among a 3 rack topology (minimum).

That way, if you loose 1 rack, then the zookeeper quorum will still be in good shape on the 2 racks still available.

Re: Zookeeper Setup for 2 Rack Data center

Explorer

I was referring to the concept of Kafka Mirroring -

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330

 

If this is implemented then can it be used to achieve rack fault tolerance across 2 racks cluster setup ? My limit is to implement scalable & fault tolerant messaging layer with 2 Rack infrastructure.

Re: Zookeeper Setup for 2 Rack Data center

Champion
Kafka Mirroring would not help. The concept behind that is to allow you to replica messages to another cluster to allow for failover between cluster, preferably in different DCs.

Both Kafka clusters, the source and destination, would need a set of ZK instances to handle the broker ids. In a two rack scenario, you would have 1 or 3 or 5 ZKs in each rack but each set would only serve that rack. A rack 1 failure would cause ZK to fail in rack 1 and Kafka would stop functioning.
Highlighted

Re: Zookeeper Setup for 2 Rack Data center

Explorer

Thanks so much for your inputs.

So is there any recommended solution (software based) to achieve rack failure fault tolerance capability with 2 racks. !! :-)

Don't have an account?
Coming from Hortonworks? Activate your account here