Kafka installation best practices in HDF

[#1]

HI,

We are going to install HDP(2.6) and HDF(2,1) in our site for now. 3 Serves in HDP and 2 servers in HDF.

Since zookeeper is not recommend to be in the same servers as Kafka and Nifi, so we are planning to have zookeeper in just HDF and Kafka in HDF servers. I have two questions regarding this architecture :

- Is the architecture the right approach to go ?

- What is the best practices for Kafka initial configuration and installationKafka.

Thanks,

SJ

[#2 · Accepted solution]

Hi @Sanaz Janbakhsh I'm not sure I understand your first question, ideally the zookeeper instances will be running on separate hardware to the Kafka nodes, however it is possible to co-locate them as long as you have enough bandwidth on the nodes, and preferably dedicate spindles to ZK.

In terms of best practices overall I'd suggest you download the slides and watch the video from this session at the recent DataWorks summit Munich: Apache Kafka Best Practices

Session video: https://www.youtube.com/watch?v=maD_7ZdyuAU

Session slides: https://www.slideshare.net/HadoopSummit/apache-kafka-best-practices

I hope this helps!