We have a 2 rack data center and both the rack has multiple servers. We are looking to setup the zookeeper nodes on these servers and it should provide failover mechanism in case of any rack failure.
Based on zookeeper admin requirement, it needs 2F+1 number of nodes and odd number of nodes in a quorum.
So in my case let say, i setup 3 zookeeper nodes on rack A and 4 on rack B. So total nodes in quorum is 7. Based on 2F+1 if rack B goes down then F is 4 here and it needs 9 nodes. If Rack A goes down then it 7. So the scenario of complete rack failure(Rack B) with with larger number of nodes will not sustain.
I want to understand if zookeeper is preferable solution to be used in case of applications with 2 rack datacenter. If yes, how can i build the infrastructure in such case.
It's not about the number of active zookeeper nodes but more about have the majority of "known" ZK nodes up.As an example the loss of one node in a 3-member ZooKeeper quorum is tolerable, because 2 out of 3 remaining machines still count as a majority (out of the fully identified quorum of 3).
Thanks for your response.
I understand zookeeper requirement about having the majority of known ZK nodes to be up. With the example you mentioned, it is fine when i have total 3 nodes and 1 node goes down, since 2F +1 = 3 and it is = our total nodes in quorum we had setup or you can also say that we still have more active nodes then failed nodes. But when we have inter rack communication(specially 2 rack), it is different. As i said, Let say -
I have active 3 ZK nodes on Rack A and 4 on rack B. In this case my quorum size is 7 and if my Rack B goes down, i am left with only 3 ZK nodes active. In this case 2F+1 says that, i should have setup 9 nodes in my cluster (F = 4) whereas i have setup 7 nodes. So zookeeper cluster across 2 Racks is not going to satisfy above rule since no of failed nodes are more then active nodes.
Is there anyways concept of Kafka mirroring or setting up isolated zookeeper cluster and then cross connect zookeeper cluster so that when rack B goes down, rack A zookeeper takes over the task help in this 2 rack communication ?