We have Hadoop cluster with the following details
3 kafka machines
3 zookeeper servers
Now the Hadoop cluster include also datanode machines And all application are using the zookeeper servers include the kafka machines
We want to do the following changes
We want to add additional 3 zookeeper servers that will be in a separate cluster
And only kafka machine will use this additional zookeeper servers
Is it possible ?
The above question were originally posted in the Community Help track. On Wed Jun 26 21:14 UTC 2019, a member of the HCC moderation staff moved it to the Design & Architecture track. The Community Help Track is intended for questions about using the HCC site itself, not technical questions about Hadoop clusters.
Is there any specific reason for having 3 separate ZK servers which belongs to a separate Cluster for Kafka ?
1. We want to add additional 3 zookeeper servers that will be in a separate cluster 2. And only kafka machine will use this additional zookeeper servers
If you are using Ambari Managed Cluster then it will not be possible via Ambari. As it will require additional efforts to explicitly update the Kafka Zookeeper settings. With this approach there will be many other issues like Ambari will notbe able to keep track of the Zookeepers running as part of a Different cluster.
So that does not look like a good idea.
Instead you can think of making the existing ZK servers more robust / high available to keep your data.
A similar discussion can be found on another HCC thread: https://community.hortonworks.com/questions/248413/should-zookeeper-be-run-on-independent-machines-i...
How to decide, how many zookeepers should I have?
@dear Jay about the additional zookeeper - I dont need the ambari management here , the reason that we want additional 3 machines of zookeeper - is because we are faced many problems in kafka , and we think part of the problem are because the current zookeeper server are installed on masters machines and masters machines are include many other services , so we want to add additional 3 zookeeper servers that will serve only the kafka machines , and the current zookeeper swerves will serv all other application , so lets summary the additional 3 zoo servers will only talk with the 3 kafka machines , and the additional 3 zoo servers will not manage under ambari , we know this but the fact we have so many problems in kafka give the idea to add additional 3 zoo servers only for the kafka , so please give your opinion about my suggestion , your opinion is so important because adding new machine are costs money and we not want to do wrong decision