Skip to main content
mike_bronson7
mike_bronson7
Gold ⭐️⭐️⭐️
June 27, 2019
Solved

Should zookeeper be run on independent machines? in production environment



we have HDP version 2.6.4 in our cluster

cluster include 285 data node machines and 3 kafka machines

for now zookeeper servers are installed on the masters machines and zookeepers are not independent machines

but since this is very important production cluster

we think to separate the zookeeper from the masters machines

and install the zookeeper servers on separate servers to be independent machines

am I right here?

jsensharma
jsensharmaAnswer
Platinum ⭐️⭐️
June 28, 2019

@Michael Bronson

Zookeeper is a light weight process hence it won't consume too much resource.

(EDIT😞 However there is the following recommendation from Hortonworks (For Kafka + Zookeeper) :

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_kafka-component-guide/content/kafka-zookeeper-multiple-apps.html

Here are several recommendations for ZooKeeper configuration with Kafka:

  • Do not run ZooKeeper on a server where Kafka is running.

  • When using ZooKeeper with Kafka you should dedicate ZooKeeper to Kafka, and not use ZooKeeper for any other components.

  • Make sure you allocate sufficient JVM memory. A good starting point is 4GB.

  • To monitor the ZooKeeper instance, use JMX metrics.

.

As far as high availability is concerned for zookeepers then you can refer to the following HCC thread which talks about more on How to decide, how many zookeepers should I have?

https://community.hortonworks.com/questions/35287/how-to-decide-how-many-zookeepers-should-i-have.html


it depends on your requirement and then based on the requirement we can decide the number of ZK hosts as 3 or 5 (..etc). More ZK comes with a cost so please go through the above thread.


View in thread

14 replies

jsensharma
Platinum ⭐️⭐️
jsensharmaAnswer
Platinum ⭐️⭐️
June 28, 2019

@Michael Bronson

Zookeeper is a light weight process hence it won't consume too much resource.

(EDIT😞 However there is the following recommendation from Hortonworks (For Kafka + Zookeeper) :

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_kafka-component-guide/content/kafka-zookeeper-multiple-apps.html

Here are several recommendations for ZooKeeper configuration with Kafka:

  • Do not run ZooKeeper on a server where Kafka is running.

  • When using ZooKeeper with Kafka you should dedicate ZooKeeper to Kafka, and not use ZooKeeper for any other components.

  • Make sure you allocate sufficient JVM memory. A good starting point is 4GB.

  • To monitor the ZooKeeper instance, use JMX metrics.

.

As far as high availability is concerned for zookeepers then you can refer to the following HCC thread which talks about more on How to decide, how many zookeepers should I have?

https://community.hortonworks.com/questions/35287/how-to-decide-how-many-zookeepers-should-i-have.html


it depends on your requirement and then based on the requirement we can decide the number of ZK hosts as 3 or 5 (..etc). More ZK comes with a cost so please go through the above thread.


Translate
    mike_bronson7
    mike_bronson7Author
    Gold ⭐️⭐️⭐️
    July 1, 2019

    @Dear Jay you said - "Kafka and Zookeepers are two services that are sensitive to disk I/O. Keeping them on same node will not be a good idea. "


    I agree


    but lets say that:


    assume we install the zookeeper on kafka machine but zookeeper will write the logs to the disk of the OS

    and not the disk of the kafka

    and we disable the swap on the kafka machine


    is this change the picture ?


    I mean in that case zookeeper not use the disk of kafka and use the disk of the OS of the kafka + disable swap

    in that case you agree ?


    Translate
    Michael-Bronson
    jsensharma
    Platinum ⭐️⭐️
    jsensharma
    Platinum ⭐️⭐️
    July 1, 2019

    @Michael Bronson

    The Doc which i shared talks about Standard Best Practice. The doc does not say that running ZK on Kafka host will not work.

    But as a best practice you should keep them on separate hosts due to load constraints.

    However, it is subject to your Pre Prod Environment Testing and Metrics Analysis on both the scenarios and then you can proceed with what suite your requirement.

    Translate
    ask_bill_brooks
    Platinum ⭐️
    ask_bill_brooks
    Platinum ⭐️
    June 28, 2019

    @Michael Bronson originally posted the above in the Community Help track. On Fri Jun 28 07:12 UTC 2019, a member of the HCC moderation staff moved it and the entire reply thread below to the Design & Architecture track. The Community Help Track is intended for questions about using the HCC site itself, not technical questions about zookeeper servers or Kafka.

    Translate
    Bill Brooks, Community Moderator Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
    Terms & ConditionsAccessibility statement