Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Zookeeper Setup for 2 Rack Data Center

Zookeeper Setup for 2 Rack Data Center

Explorer

Hi,

We have a 2 rack data center and both the rack has multiple servers. We are looking to setup the zookeeper nodes on these servers and it should provide failover mechanism in case of any rack failure.

Based on zookeeper admin requirement, it needs 2F+1 number of nodes and odd number of nodes in a quorum.

So in my case let say, i setup 3 zookeeper nodes on rack A and 4 on rack B. So total nodes in quorum is 7. Based on 2F+1 if rack B goes down then F is 4 here and it needs 9 nodes. If Rack A goes down then it 7. So the scenario of complete rack failure(Rack B) with with larger number of nodes will not sustain.

I want to understand if zookeeper is preferable solution to be used in case of applications with 2 rack datacenter. If yes, how can i build the infrastructure in such case.

2 REPLIES 2
Highlighted

Re: Zookeeper Setup for 2 Rack Data Center

New Contributor

It's not about the number of active zookeeper nodes but more about have the majority of "known" ZK nodes up.

As an example the loss of one node in a 3-member ZooKeeper quorum is tolerable, because 2 out of 3 remaining machines still count as a majority (out of the fully identified quorum of 3).
  • No loss of functionality will be experienced with the loss of only one of the peers (in three), two peers (in five), etc.
  • One of the remaining two will automatically be assigned the leader role, in case of a leader failure.
  • Writes can continue to happen, and no special mode is invoked.
  • Clients will continue to see the same behaviour they expect out ofZooKeeper, even in such a situation.
  • No manual intervention is required for this procedure, ZK is automatically HA.
The administrator's guide of ZooKeeper covers this:

http://archive.cloudera.com/cdh5/cdh/5/zookeeper/zookeeperAdmin.html

Re: Zookeeper Setup for 2 Rack Data Center

Explorer

Hi,

Thanks for your response.

I understand zookeeper requirement about having the majority of known ZK nodes to be up. With the example you mentioned, it is fine when i have total 3 nodes and 1 node goes down, since 2F +1 = 3 and it is = our total nodes in quorum we had setup or you can also say that we still have more active nodes then failed nodes. But when we have inter rack communication(specially 2 rack), it is different. As i said, Let say -

I have active 3 ZK nodes on Rack A and 4 on rack B. In this case my quorum size is 7 and if my Rack B goes down, i am left with only 3 ZK nodes active. In this case 2F+1 says that, i should have setup 9 nodes in my cluster (F = 4) whereas i have setup 7 nodes. So zookeeper cluster across 2 Racks is not going to satisfy above rule since no of failed nodes are more then active nodes.

Is there anyways concept of Kafka mirroring or setting up isolated zookeeper cluster and then cross connect zookeeper cluster so that when rack B goes down, rack A zookeeper takes over the task help in this 2 rack communication ?