There is no such rule for Kafka Brokers. Zookeeper should maintain a quorum or (n/2 + 1) total machines (of n) that agree on leader-election values and locks, that results in a total odd number to accommodate for hardware and network failure scenarios. From "Kafka - The Definitive Guide", as well as Apache Zookeeper site, you generally will have negative side effects from having more than 5 or 7 Zookeeper servers total serving applications using it. You should have more than 3 Zookeepers because if one goes down, you are only left with 2, which results in that "split brain". With 5 servers, two can go down, and you still have 2 servers + 1 available for the "tie breaker" vote. For 7, you can loose up to 4 zookeepers and still be good. ... View more

Contrary to answer by @Harshali Patel, exhaustion is not defined as an uneven distribution, it is rather a cause of it. A datanode has a property that you can set which defines a threshold of data must be reserved for the OS on that server. Once that limit is exceeded, the datanode process will stop and log an error telling you to delete some files from it. HDFS will continue to function with the other datanodes. The balancer can be ran to keep storage space healthy and even. ... View more

TaskTracker & JobTracker doesn't exist with YARN. The default replication factor is 3. ... View more

Please see previous question - https://community.hortonworks.com/questions/167618/how-to-specify-more-than-one-path-for-the-storage.html ... View more

There is no such support for renaming https://issues.apache.org/jira/browse/KAFKA-2333 If you want to clone, then use MirrorMaker https://community.hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html ... View more

@Michael Bronson Kafka stores the latest offsets in memory before they are sent to disk, therefore, the more memory the better, with a max of 8G. And I would assume that the heap properties can be set from Ambari rather than individually on the broker, but I don't use Kafka from HDP, so I can't say. ... View more

The recommendation here would be to increase the heap space allocated to the Kafka process or reduce the amount of other processes running on the same server. For example, in a production environment, the Kafka brokers should be standalone servers -- not on the same hardware as Zookeeper or other Hadoop processes. ... View more